Most local AI apps, including Ollama, LM Studio, llama.cpp and KoboldCpp, run models in the GGUF format. Popular models usually have ready-made GGUF files on Hugging Face, but not always. A brand new release, a fine-tune you made yourself or a niche model may only exist as the original Hugging Face safetensors.
The fix is to convert it yourself with llama.cpp. You turn the Hugging Face folder into a 16-bit GGUF, then shrink it with llama-quantize to a size that fits your GPU, such as Q4_K_M.
This guide covers the whole process: setup, conversion, building llama-quantize, choosing a quant type, using an importance matrix, testing the result, running it in Ollama or LM Studio, and fixing common errors. Commands follow the official llama.cpp README as of November 2026.
- Get llama.cpp:
git clone https://github.com/ggml-org/llama.cppthenpip install -r requirements.txt. - Convert:
python convert_hf_to_gguf.py ./my-model --outfile my-model-bf16.gguf --outtype bf16 - Build:
cmake -B buildthencmake --build build --config Release. - Quantize:
./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M - Good defaults: Q4_K_M for most GPUs, Q5_K_M or Q6_K if you have room, Q8_0 for near-lossless.
- No setup option: the GGUF-my-repo space on Hugging Face does it online.
What GGUF and Quantization Are
GGUF is a single-file model format used by llama.cpp and the apps built on it. It holds the weights, tokenizer and model settings together, so one file is all you need.
Quantization stores each weight with fewer bits. A 16-bit model uses about 16 bits per weight; Q4_K_M uses just under 5 on average. That makes the file roughly a third of the size with only a small quality loss, so it fits in far less VRAM.
The llama.cpp README gives a clear example for Llama 3.1 at Q4_K_M:
| Model | Original size | Q4_K_M size |
|---|---|---|
| 8B | 32.1 GB | 4.9 GB |
| 70B | 280.9 GB | 43.1 GB |
| 405B | 1,625.1 GB | 249.1 GB |
The README notes that disk and memory needs are about the same, since the model is loaded fully.
What You Need
- Python 3 and Git.
- CMake and a C++ compiler (Visual Studio Build Tools on Windows, Xcode tools on Mac, build-essential on Linux) to build llama-quantize.
- Disk space for the original model, the 16-bit GGUF and the quantized file. Plan for roughly three times the original size during the process.
- The model folder from Hugging Face, with its config.json, tokenizer files and safetensors. See how to download models from Hugging Face.
- A supported architecture. The converter reads config.json to detect it, and very new architectures need an up-to-date llama.cpp.
Set Up llama.cpp
- Clone the repo:
git clone https://github.com/ggml-org/llama.cpp cd llama.cpp - Create a virtual environment (recommended):
python3 -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate - Install the Python requirements:
python3 -m pip install -r requirements.txt - Update transformers if needed. The README says requirements pin transformers 4 for compatibility, but some newer models need transformers 5:
pip install -U transformers.
Convert the Model to GGUF
Point the converter at the local model folder:
python convert_hf_to_gguf.py ./my-model --outfile my-model-bf16.gguf --outtype bf16
Or let it download straight from Hugging Face with --remote:
python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
| –outtype | When to use |
|---|---|
bf16 |
Most modern models, which are trained in bfloat16 |
f16 |
Older models released in float16 |
auto (or omit it) |
The README says this also works well for 16-bit models |
f32 |
Rarely needed; very large files |
Vision models: convert the projector separately with --mmproj. The README keeps mmproj files at bf16 or q8_0:
python convert_hf_to_gguf.py --mmproj --outfile mmproj-gemma-4-E2B-it-Q8_0.gguf --outtype q8_0 --remote google/gemma-4-E2B-it
LoRA adapters have their own script: python convert_lora_to_gguf.py ./my-lora.
Build llama-quantize
llama-quantize is a compiled tool, so build llama.cpp once from the repo folder:
cmake -B build
cmake --build build --config Release
The tools land in build/bin/ (on Windows with Visual Studio, often build\bin\Release\). Add GPU flags such as -DGGML_CUDA=ON to the first command if you also want fast GPU inference for testing; quantizing itself doesn’t need the GPU. Prebuilt binaries from the llama.cpp releases page work too.
Quantize to Q4_K_M and Others
./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M
- Input: your 16-bit GGUF.
- Output: any file name you like; include the quant type so you can tell files apart.
- Type: the quant preset. Names aren’t case-sensitive.
- Repeat for each size you want, always starting from the 16-bit file.
| Useful option | What it does |
|---|---|
--imatrix file |
Use an importance matrix for better quality |
--leave-output-tensor |
Keep output.weight unquantized (bigger, maybe better) |
--output-tensor-type, --token-embedding-type |
Set specific types for those tensors |
--pure |
Disable k-quant mixtures; one type for all tensors |
--keep-split |
Keep the same shards as a split input |
--allow-requantize |
Requantize an already-quantized file (the README warns this can severely reduce quality) |
Which Quant Type to Choose
Sizes for Llama 3.1 8B from the llama.cpp README:
| Type | Bits per weight | Size (GiB) | Use it when |
|---|---|---|---|
| Q8_0 | 8.50 | 7.95 | You want near-original quality and have the memory |
| Q6_K | 6.56 | 6.14 | Very high quality, still much smaller |
| Q5_K_M | 5.70 | 5.33 | A step up from Q4 with a little more VRAM |
| Q4_K_M | 4.89 | 4.58 | The usual sweet spot for size and quality |
| IQ4_XS | 4.46 | 4.17 | Slightly smaller than Q4_K_M; best with an imatrix |
| Q3_K_M | 4.00 | 3.74 | Tight on memory |
| IQ2_M | 2.93 | 2.74 | Huge models on small GPUs; needs an imatrix |
Qwen’s llama.cpp guide says Q8_0, Q5_K_M and Q4_K_M are common choices for 8B models. A rule of thumb: pick the largest quant that fits fully in your GPU with room left for context. Our RTX 5070 for AI guide shows what fits in 12 GB.
Better Small Quants With an Importance Matrix
An importance matrix tells the quantizer which weights matter most, using sample text. It helps a lot at 4 bits and below, and llama-quantize warns if you make 1-bit or 2-bit quants without one.
./build/bin/llama-imatrix -m my-model-bf16.gguf -f calibration.txt --chunk 512 -o my-model-imatrix.dat -ngl 80
./build/bin/llama-quantize --imatrix my-model-imatrix.dat my-model-bf16.gguf my-model-IQ4_XS.gguf IQ4_XS
- calibration.txt should be text like your real use: chat, code or your language.
- -ngl offloads layers to the GPU to speed it up.
Test Your GGUF
- Quick chat test:
./build/bin/llama-cli -m my-model-Q4_K_M.gguf -p "Explain GGUF in one sentence." -n 64Coherent output means the conversion worked.
- Compare quality with perplexity on a test file (lower is better):
./build/bin/llama-perplexity -m my-model-Q4_K_M.gguf -f wiki.test.raw -ngl 80Qwen notes Wikipedia text isn’t a great test for chat models, so also try your own prompts.
- Serve it as an API with llama-server: see our llama-server guide.
Run It in Ollama or LM Studio
- Ollama: create a Modelfile with
FROM ./my-model-Q4_K_M.gguf, thenollama create my-model -f Modelfile. Full details in our Ollama Modelfile guide. If you upload the GGUF to Hugging Face, you can also run it from hf.co in Ollama. - LM Studio: import the file into your models folder; see how to use LM Studio.
- Not sure which app? Read LM Studio vs Ollama.
The No-Setup Option: GGUF-my-repo
The llama.cpp README points to the GGUF-my-repo space on Hugging Face (ggml-org/gguf-my-repo). Sign in, enter a model repo, choose quant types, and it builds the GGUF files and publishes them to your account. It syncs with llama.cpp’s main branch every 6 hours. It’s ideal for public models when you don’t want to install anything.
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| “Model architecture not supported” | Old llama.cpp or a brand new architecture | git pull, reinstall requirements, retry; otherwise wait for support |
| Transformers or tokenizer error | Model needs a newer transformers | pip install -U transformers |
| Missing tokenizer files | Incomplete download | Download the full repo, including tokenizer files |
| Killed or out of memory during conversion | Not enough RAM | Close apps, add swap, or use GGUF-my-repo |
| llama-quantize not found | Not built, or wrong path | Build with CMake; check build/bin or build\bin\Release |
| Output is gibberish | Wrong chat template or broken conversion | Test with llama-cli; update llama.cpp and reconvert |
| Quality much worse than expected | Requantized from a quantized file | Always quantize from the 16-bit GGUF |
| Vision model can’t see images | No mmproj file | Convert with –mmproj and load both files |
FAQ
How do I convert a Hugging Face model to GGUF?
Clone llama.cpp, install its requirements, run convert_hf_to_gguf.py on the model folder, then shrink it with llama-quantize.
Which quantization is best?
Q4_K_M is the usual balance of size and quality. Use Q5_K_M, Q6_K or Q8_0 if you have more memory.
Do I need a GPU to convert a model to GGUF?
No. Conversion and quantization run on the CPU. A GPU only speeds up imatrix creation and testing.
Should I use bf16 or f16 when converting?
Use bf16 for most modern models trained in bfloat16, or leave –outtype on auto.
Can I convert a LoRA to GGUF?
Yes, with convert_lora_to_gguf.py in the llama.cpp repo.
Is there an online GGUF converter?
Yes. The GGUF-my-repo space on Hugging Face converts and quantizes public models without any setup.
Can I requantize a Q8 file to Q4?
Only with –allow-requantize, and the README warns quality can drop severely. Start from 16-bit instead.




