Few errors are as common in local AI as this one: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 15.92 GiB of which 1.10 GiB is free. It means the job asked your graphics card for more memory than it had free. The good news is that it’s almost always fixable without buying a new GPU. In this guide I explain how to read the message, find what is using your VRAM, and apply the fixes that work best for PyTorch scripts, ComfyUI, Ollama, llama.cpp, LM Studio and LoRA training, in the order I try them myself on my 16 GB RTX 5080s.
- First: close other GPU apps and kill leftover Python or llama-server processes (
nvidia-smishows them). - Images and video (ComfyUI): use FP8 or GGUF models, lower resolution, frames or batch size, and add
--reserve-vram 1; try--lowvramas a last resort. - LLMs (Ollama, llama.cpp, LM Studio): use a smaller quant (Q4_K_M), shorten the context, quantise the KV cache, or offload some layers to the CPU.
- PyTorch and training: lower batch size, use gradient checkpointing, BF16/FP16, 8-bit optimisers or QLoRA, and set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueto reduce fragmentation. - Slow instead of OOM? Windows may be spilling into shared memory. Set Sysmem Fallback to “Prefer No Sysmem Fallback” to see real errors.
How to Read the Error Message
A typical PyTorch message has four useful numbers:
- Tried to allocate X GiB: the single block the job needed at that moment.
- Total capacity: your card’s VRAM (a 16 GB card shows about 15.9 GiB).
- Free: what was left when it failed.
- Reserved but unallocated: memory PyTorch holds but couldn’t use as one block. If this is large (1 GB or more), you have fragmentation, and the
expandable_segmentsfix below often solves it.
Other tools phrase it differently: llama.cpp prints “failed to allocate buffer” or “cudaMalloc failed: out of memory”, Ollama may say “model requires more system memory” or silently fall back to CPU, and ComfyUI shows “Allocation on device” errors in the console.
Step 1: Find What Is Using VRAM
- Run
nvidia-smi. Check memory used vs total and the process list. - On Windows, open Task Manager > Details, add the Dedicated GPU memory column and sort by it.
- Look for leftovers: an old
python.exe,llama-server,ollamawith a model still loaded (ollama ps), or a second ComfyUI instance. - Browsers with hardware acceleration, Discord, OBS, game launchers and wallpaper apps can each hold 300 MB to 2 GB.
Full commands for every platform are in how to check GPU VRAM and usage.
Step 2: Quick Fixes for Everyone
- Close or kill the processes you found. On Linux:
kill PID. In Ollama:ollama stop modelname. - Restart the app (or the Python kernel in a notebook) to clear fragmented memory.
- Use a smaller version of the model: FP8 instead of BF16, Q4_K_M instead of Q8.
- Do less at once: lower batch size, resolution, frame count or context length.
- Move the display to integrated graphics if your CPU has it, freeing 300-800 MB on the GPU.
Fixes for ComfyUI
- Use FP8 or GGUF models. A 20B image model in BF16 is about 40 GB; FP8 halves it, GGUF Q4-Q6 shrinks it further. See the ComfyUI-GGUF guide and which GGUF quant to download.
- Use an FP8 or GGUF text encoder too. Big encoders like umt5-xxl or Qwen VL can take 5-10 GB on their own.
- Launch flags:
--reserve-vram 1keeps 1 GB free for Windows;--lowvramoffloads more aggressively (slower);--disable-smart-memoryunloads models between runs. Details in ComfyUI command line arguments. - Tiled VAE decode: swap VAE Decode for VAE Decode (Tiled). Decoding high-res images and video frames is a frequent OOM point.
- Video: fewer frames, lower resolution, and block swapping nodes (offloading transformer blocks to RAM) in Wan and similar workflows.
- Attention: SageAttention reduces memory and speeds up many models; see installing SageAttention on Windows.
- Two GPUs: put the text encoder or VAE on the second card with multi-GPU loader nodes.
More ComfyUI-specific errors are collected in ComfyUI errors and fixes.
Fixes for Ollama, llama.cpp and LM Studio
| Fix | Ollama | llama.cpp | LM Studio |
|---|---|---|---|
| Smaller quant | ollama run model:14b-q4_K_M style tags |
Download a Q4_K_M or Q3_K_M GGUF | Pick a smaller file in the model browser |
| Shorter context | OLLAMA_CONTEXT_LENGTH=8192 or /set parameter num_ctx 8192 |
-c 8192 |
Context Length slider |
| Quantised KV cache | OLLAMA_FLASH_ATTENTION=1 + OLLAMA_KV_CACHE_TYPE=q8_0 |
-fa on -ctk q8_0 -ctv q8_0 |
KV cache quantisation options |
| Offload layers to CPU | Automatic | Lower -ngl |
GPU Offload slider |
| MoE experts in RAM | Automatic for some models | --n-cpu-moe N or -ot rules |
Offload experts option |
| One model at a time | OLLAMA_MAX_LOADED_MODELS=1 |
– | Eject other models |
Context is the hidden VRAM eater: a 14B model with a 128K context can need more memory for the KV cache than for the weights. Start with 8K-16K and raise it only when you need it. Which models fit which cards: best Ollama models by VRAM.
Fixes for PyTorch Scripts and Training
- Lower batch size and use gradient accumulation to keep the effective batch.
- Mixed precision:
torch.autocast("cuda", dtype=torch.bfloat16)(FP16 on older cards like the T4). - Gradient checkpointing:
model.gradient_checkpointing_enable()in Hugging Face models trades compute for a large memory saving. - 8-bit or paged optimisers (bitsandbytes AdamW8bit) and QLoRA (4-bit base model) for fine-tuning; Unsloth makes this easy.
- Inference without gradients: wrap with
torch.inference_mode(). - Fragmentation: set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truebefore starting Python (on Windows:set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True). - Free memory between steps: delete large tensors with
del, thentorch.cuda.empty_cache(). This returns cached memory to the driver but doesn’t fix leaks. - Check for leaks: storing loss tensors in a list keeps the graph alive; store
loss.item()instead.
Windows Shared Memory and Sysmem Fallback
On Windows, NVIDIA drivers can spill into system RAM (shown as “Shared GPU memory” in Task Manager) instead of throwing an error. The job keeps running but can become 5-20x slower. If you’d rather know immediately:
- Open NVIDIA Control Panel > Manage 3D settings.
- Find CUDA – Sysmem Fallback Policy.
- Set it to Prefer No Sysmem Fallback globally or for python.exe.
Now you’ll get a clear OOM error and can apply a fix above, instead of waiting an hour for a crawling render.
Which Fix Saves the Most Memory
| Fix | Typical saving | Cost |
|---|---|---|
| BF16 to FP8 model | About 50% of weights | Tiny quality loss |
| Q8 to Q4_K_M GGUF | About 40% of weights | Small quality loss |
| Halve context length | About 50% of KV cache | Less memory for long chats |
| KV cache q8_0 | About 50% of KV cache | Minimal |
| Gradient checkpointing | Large share of activations | About 20-30% slower training |
| Halve batch size | About 50% of activations | Slower or noisier training |
| Tiled VAE decode | Several GB at high resolution | Slightly slower decode |
| CPU offload / lowvram | As much as needed | Much slower |
Savings are approximate and depend on the model.
When You Really Need More VRAM
If you’ve tried everything and the model still won’t fit, it’s time to rent or upgrade. Renting a 24-96 GB GPU for an hour is cheap; see running ComfyUI on RunPod. For buying advice, read how much VRAM do I need and RTX 5090 vs RTX 5080.
Odd Cases
OOM with plenty of free VRAM shown
Fragmentation: the free memory isn’t one big block. Use expandable_segments:True or restart the process.
OOM only on the second run
Something from the first run stayed loaded. In ComfyUI click Unload models / Free memory, or use --disable-smart-memory. In notebooks, restart the kernel.
Two GPUs but it still fails on GPU 0
The app uses the first GPU by default. Set CUDA_VISIBLE_DEVICES=1 or the app’s device option to use the other card, or split the model with the tool’s multi-GPU settings.
“CUDA error: out of memory” right at startup
Another process has grabbed the GPU, or a driver issue is reporting the wrong capacity. Reboot, then check nvidia-smi before launching.
System RAM runs out instead
Offloading moves the load to RAM. Large MoE models with CPU experts need 64-128 GB of system RAM. Close other apps or use a smaller model.
Frequently Asked Questions
What does CUDA out of memory mean?
Your program asked the GPU for more memory than was free. It usually means the model, batch, resolution or context is too big for your VRAM, or other apps are using some of it.
How do I fix CUDA out of memory in ComfyUI?
Use FP8 or GGUF models and text encoders, lower resolution, frames or batch size, use VAE Decode (Tiled), and launch with –reserve-vram 1 or –lowvram if needed.
Does torch.cuda.empty_cache() fix out of memory errors?
Only sometimes. It releases cached memory back to the driver but doesn’t free tensors you still reference or fix the cause. Delete big tensors first, and reduce batch size or precision.
What is PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True?
A PyTorch setting that reduces memory fragmentation, which helps when the error shows lots of reserved but unallocated memory.
How do I stop Ollama running out of GPU memory?
Use a smaller quant, reduce the context with OLLAMA_CONTEXT_LENGTH, enable flash attention with a q8_0 KV cache, and keep only one model loaded.
Why is my AI job slow instead of crashing with out of memory?
On Windows, the driver may be spilling into shared system memory. Set CUDA Sysmem Fallback Policy to Prefer No Sysmem Fallback in the NVIDIA Control Panel to get a clear error instead.
How much VRAM do I need to avoid out of memory errors?
It depends on the model. 16 GB handles most image models in FP8 and 14B-20B LLMs; 24-32 GB is better for video and 27B-32B LLMs.
Final Thoughts
My order is always the same: check nvidia-smi for leftovers, switch to a smaller model format, cut the batch, resolution or context, then reach for offloading. That sequence fixes the vast majority of out-of-memory errors on 12-16 GB cards, and the only real cost is a little quality or speed.
Keep these handy: ComfyUI Reconnecting error fix, Ollama not using GPU fix and the Ollama commands cheat sheet.