GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q8_0 and Which File to Download for Your VRAM

Open any model page on Hugging Face and you get a wall of files: Q2_K, IQ3_XS, Q4_K_M, Q5_K_S, Q6_K, Q8_0, F16, sometimes with “UD” or “i1” added. Pick too big and the model will not load or crawls along in system RAM. Pick too small and the answers get noticeably worse. This guide explains what every GGUF name means, how much quality each level keeps, how to work out the file size and VRAM you need before you download, and which file to pick for 8, 12, 16, 24 and 32 GB GPUs. It covers both local LLMs (llama.cpp, Ollama, LM Studio) and GGUF image and video models in ComfyUI.

What Is GGUF and What Is Quantization?

GGUF is a single-file model format created by the llama.cpp project. One .gguf file holds the weights plus everything needed to run them: the tokenizer, the chat template and the model settings. It is the standard format for llama.cpp, Ollama, LM Studio, Jan and many other local AI apps, and the ComfyUI-GGUF node pack brings it to image and video models.

Quantization stores each weight with fewer bits. Models are trained in 16-bit (F16 or BF16). Storing them in about 4 bits makes the file roughly a quarter of the size, so it fits in less VRAM and runs faster, at the cost of a small loss in accuracy. GGUF does not round every weight blindly: weights are grouped into small blocks that share a scale, and important layers can keep more bits than others.

How to Read a GGUF File Name

Take Qwen3-14B-Q4_K_M.gguf:

• Q4: about 4 bits per weight on average (the number is the main bit level).
• K: a “k-quant”, the modern block format with super-blocks and better accuracy than the old formats.
• M: the size variant. S (small), M (medium) and L (large) decide how many sensitive tensors get extra bits. M is the usual sweet spot.

Other names you will see:

• IQ2_XXS, IQ3_XS, IQ4_XS and similar: “i-quants”, a newer family that keeps more quality at very low bit levels.
• Q4_0, Q4_1, Q5_0, Q5_1: legacy formats. Simpler, sometimes faster on CPUs and ARM chips, but less accurate per bit than k-quants.
• Q8_0: 8-bit. Almost identical to the original model.
• F16, BF16, F32: unquantized.
• -00001-of-00003.gguf: a model split into parts. Download all parts into one folder and point the app at the first one.

Every Quant Type Compared

Bits per weight (bpw) are approximate figures from llama.cpp for typical models. Real files vary a little because some tensors, such as embeddings, are stored at higher precision.

How to Calculate File Size

You can estimate any GGUF file before you download it:

file size (GB) = parameters (billions) x bits per weight / 8

Examples:

• 8B at Q4_K_M: 8 x 4.85 / 8 = about 4.9 GB
• 14B at Q4_K_M: 14 x 4.85 / 8 = about 8.5 GB
• 32B at Q4_K_M: 32 x 4.85 / 8 = about 19.4 GB
• 70B at Q4_K_M: 70 x 4.85 / 8 = about 42 GB

Real-world check: Liquid AI’s LFM3-1.7B is 3.47 GB at F16, 1.85 GB at Q8_0 and 1.1 GB at Q4_K_M, which lines up with the formula (small models run a little larger at Q4 because their embeddings are a bigger share of the file). See our LFM3 guide for that model.

For Mixture-of-Experts models, use the total parameter count for file size, not the active count. A model with 30B total and 3B active is still a 30B-size file; the small active count only makes it faster.

How Much VRAM You Really Need (KV Cache)

The file is only part of it. The model also needs memory for the KV cache, which stores the conversation so far, plus a little working space:

VRAM needed = file size + KV cache + about 0.5 to 1 GB overhead

The KV cache grows with context length. For a typical modern 8B model with grouped-query attention (32 layers, 8 KV heads, head size 128), the cache in 16-bit is about 128 KB per token:

Bigger models have more layers and a bigger cache. Some new architectures (hybrid, sliding-window or linear attention) need far less. The formula, if you want to check a specific model’s config:

KV bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_value

Two easy ways to save memory:

• Do not set a bigger context than you need. 8K to 16K is plenty for most chats.
• Quantize the KV cache to q8_0. In llama.cpp, add -fa on -ctk q8_0 -ctv q8_0. In Ollama, set the environment variables OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0. The quality cost is very small.

Which Quant for Your GPU

These picks keep the whole model on the GPU with a moderate (8K to 16K) context. They are calculated from the formula above, so treat them as starting points, not guarantees.

Not enough VRAM? llama.cpp and Ollama can split a model between GPU and system RAM. It works, but every layer on the CPU slows generation a lot. MoE models are the exception: keeping the experts in system RAM and the rest on the GPU (llama.cpp --n-cpu-moe) often stays usable. Our MiniCPM-V 4.7 guide shows that trick in practice.

K-Quants vs I-Quants vs Legacy Quants

Simple rule: at 4 bits and above, use K-quants (Q4_K_M, Q5_K_M, Q6_K) or Q8_0. Below 4 bits, use I-quants (IQ3_XS, IQ2_M).

imatrix, “UD” and Other Tags

• imatrix (i1, imat): an importance matrix is measured by running sample text through the model. The quantizer then spends more bits where they matter. It helps most at low bit levels. Many uploaders, such as bartowski and mradermacher, publish imatrix quants by default.
• UD (Unsloth Dynamic): Unsloth’s quants choose different bit levels per layer. A UD-Q4_K_XL file is a little larger than Q4_K_M but usually more accurate.
• Official vs community quants: some labs (for example Liquid AI and Qwen) publish their own GGUFs. These often include the right chat template and tested settings, so prefer them when they exist.
• QAT: “quantization-aware training” models were trained to be quantized and lose less at 4 bits than a normal model.

Bigger Model at Low Quant, or Smaller Model at High Quant?

A common question: is a 14B model at Q3 better than an 8B model at Q8? A widely used rule of thumb from the llama.cpp community:

• Down to about 4 bits, the bigger model usually wins.
• Around 3 bits it is close; test both on your own tasks.
• At 2 bits and below, the big model often falls apart, and the smaller model at Q4 to Q6 is the safer pick.

Tasks that need precision, such as coding, maths and tool calling, suffer from quantization first. For those, step up one level (Q5_K_M or Q6_K) when you can.

How to Download Just One File

Do not clone a whole GGUF repo; it can hold 20 files and hundreds of GB. Download only the quant you want:

pip install -U "huggingface_hub[cli]"
huggingface-cli download bartowski/SOME-MODEL-GGUF --include "*Q4_K_M*" --local-dir ./models

Replace the repo name with the model you want. For split files, the --include pattern grabs all parts. In LM Studio, the search screen lists every quant and marks which ones fit fully on your GPU.

Run It: llama.cpp, Ollama, LM Studio

Ollama

Ollama can pull a specific quant straight from Hugging Face:

ollama run hf.co/USER/REPO-GGUF:Q4_K_M

For a local file, create a Modelfile containing FROM ./model-Q4_K_M.gguf, then run ollama create mymodel -f Modelfile and ollama run mymodel.

llama.cpp

llama-server -m ./models/model-Q4_K_M.gguf -ngl 99 -c 16384 -fa on -ctk q8_0 -ctv q8_0

• -ngl 99 puts all layers on the GPU (lower it if you run out of VRAM).
• -c 16384 sets the context length.
• -fa on -ctk q8_0 -ctv q8_0 turns on flash attention and halves the KV cache.
• For MoE models, add --n-cpu-moe 10 (raise the number until it fits) to keep some experts in system RAM.

LM Studio

Search for the model, pick the quant marked as fitting your GPU, and use the GPU offload slider in the model settings. LM Studio shows estimated memory use before you load.

For worked examples, see our guides to Nemotron 3.5 Bolt E4B and LFM3.

GGUF for Image and Video Models in ComfyUI

The same quant names apply to diffusion models, but image and video models are more sensitive than LLMs. A slightly wrong weight in an LLM rarely changes the next word; in an image model it can shift colours, soften fine detail or garble text.

To use them, install the ComfyUI-GGUF pack:

cd ComfyUI/custom_nodes
git clone https://github.com/city96/ComfyUI-GGUF
pip install -r ComfyUI-GGUF/requirements.txt

Then put GGUF diffusion models in ComfyUI/models/unet/ and load them with Unet Loader (GGUF). GGUF text encoders go in ComfyUI/models/text_encoders/ (or ComfyUI/models/clip/) with CLIPLoader (GGUF). The VAE stays a normal safetensors file in ComfyUI/models/vae/. Our ComfyUI model folder cheat sheet has the full layout, and the MiniMax H3 GGUF workflow is a real Q4_K_M video example.

In ComfyUI, GGUF is not always faster than fp8. On recent NVIDIA cards, fp8 or int8 files can run quicker because the GPU supports those formats directly, while GGUF weights are unpacked on the fly. Choose GGUF when it lets you fit a higher-quality file, not for speed alone.

FAQ

Which GGUF should I download if I am not sure?

Q4_K_M. If it fits with room to spare, try Q5_K_M or Q6_K.

Is Q8_0 worth it over Q6_K?

For most chat use, the difference is tiny. Q8_0 is worth it for small models (under about 4B) and for image or video models.

What is the difference between Q4_K_S and Q4_K_M?

Q4_K_M gives more bits to the most sensitive tensors, so it is slightly larger and slightly more accurate. Pick M unless you need the few hundred MB.

Why is my model slow even though it loaded?

Part of it is probably in system RAM. Check the load log for how many layers went to the GPU, then use a smaller quant or a shorter context.

Can I convert a model to GGUF myself?

Yes. llama.cpp includes convert_hf_to_gguf.py to make an F16 GGUF and llama-quantize to make smaller quants from it. Most popular models already have community GGUFs, so check Hugging Face first.

Do GGUF files work on Mac?

Yes. llama.cpp, Ollama and LM Studio all run GGUF on Apple Silicon using Metal, and the unified memory means larger models fit than on a PC GPU with the same “VRAM”.

Final Thoughts

Choosing a GGUF comes down to three numbers: the parameter count, the bits per weight and your VRAM. Multiply the first two and divide by eight for the file size, add the KV cache for your context, and pick the highest quant that fits. When in doubt, Q4_K_M is the safe default; step up to Q5 or Q6 for coding and image models, and step down to IQ quants only when you need a bigger model to fit.

More local AI guides:

• Liquid AI LFM3: How to Run It Locally
• Nemotron 3.5 Bolt E4B: Which GGUF to Download
• ComfyUI Model Folder Cheat Sheet
• ComfyUI Errors and Fixes

Sources

• GitHub: ggml-org/llama.cpp (quant types, server options, conversion tools)
• Hugging Face docs: GGUF
• Ollama FAQ (flash attention and KV cache type)
• GitHub: city96/ComfyUI-GGUF