Skip to content

How to Convert a Hugging Face Model to GGUF (2026): llama.cpp and Quantize Guide

6–9 minutes
Laughing woman with a braid in a mustard blazer in a podcast studio holding a card reading GGUF, with the text Make Your Own GGUF

Most local AI apps, including Ollama, LM Studio, llama.cpp and KoboldCpp, run models in the GGUF format. Popular models usually have ready-made GGUF files on Hugging Face, but not always. A brand new release, a fine-tune you made yourself or a niche model may only exist as the original Hugging Face safetensors.

The fix is to convert it yourself with llama.cpp. You turn the Hugging Face folder into a 16-bit GGUF, then shrink it with llama-quantize to a size that fits your GPU, such as Q4_K_M.

This guide covers the whole process: setup, conversion, building llama-quantize, choosing a quant type, using an importance matrix, testing the result, running it in Ollama or LM Studio, and fixing common errors. Commands follow the official llama.cpp README as of November 2026.

TL;DR

  • Get llama.cpp: git clone https://github.com/ggml-org/llama.cpp then pip install -r requirements.txt.
  • Convert: python convert_hf_to_gguf.py ./my-model --outfile my-model-bf16.gguf --outtype bf16
  • Build: cmake -B build then cmake --build build --config Release.
  • Quantize: ./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M
  • Good defaults: Q4_K_M for most GPUs, Q5_K_M or Q6_K if you have room, Q8_0 for near-lossless.
  • No setup option: the GGUF-my-repo space on Hugging Face does it online.

What GGUF and Quantization Are

GGUF is a single-file model format used by llama.cpp and the apps built on it. It holds the weights, tokenizer and model settings together, so one file is all you need.

Quantization stores each weight with fewer bits. A 16-bit model uses about 16 bits per weight; Q4_K_M uses just under 5 on average. That makes the file roughly a third of the size with only a small quality loss, so it fits in far less VRAM.

The llama.cpp README gives a clear example for Llama 3.1 at Q4_K_M:

Model Original size Q4_K_M size
8B 32.1 GB 4.9 GB
70B 280.9 GB 43.1 GB
405B 1,625.1 GB 249.1 GB

The README notes that disk and memory needs are about the same, since the model is loaded fully.

What You Need

  • Python 3 and Git.
  • CMake and a C++ compiler (Visual Studio Build Tools on Windows, Xcode tools on Mac, build-essential on Linux) to build llama-quantize.
  • Disk space for the original model, the 16-bit GGUF and the quantized file. Plan for roughly three times the original size during the process.
  • The model folder from Hugging Face, with its config.json, tokenizer files and safetensors. See how to download models from Hugging Face.
  • A supported architecture. The converter reads config.json to detect it, and very new architectures need an up-to-date llama.cpp.

Set Up llama.cpp

  1. Clone the repo:
    git clone https://github.com/ggml-org/llama.cpp
    cd llama.cpp
  2. Create a virtual environment (recommended):
    python3 -m venv .venv
    source .venv/bin/activate   # Windows: .venv\Scripts\activate
  3. Install the Python requirements:
    python3 -m pip install -r requirements.txt
  4. Update transformers if needed. The README says requirements pin transformers 4 for compatibility, but some newer models need transformers 5: pip install -U transformers.

Convert the Model to GGUF

Point the converter at the local model folder:

python convert_hf_to_gguf.py ./my-model --outfile my-model-bf16.gguf --outtype bf16

Or let it download straight from Hugging Face with --remote:

python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
–outtype When to use
bf16 Most modern models, which are trained in bfloat16
f16 Older models released in float16
auto (or omit it) The README says this also works well for 16-bit models
f32 Rarely needed; very large files

Vision models: convert the projector separately with --mmproj. The README keeps mmproj files at bf16 or q8_0:

python convert_hf_to_gguf.py --mmproj --outfile mmproj-gemma-4-E2B-it-Q8_0.gguf --outtype q8_0 --remote google/gemma-4-E2B-it

LoRA adapters have their own script: python convert_lora_to_gguf.py ./my-lora.

Build llama-quantize

llama-quantize is a compiled tool, so build llama.cpp once from the repo folder:

cmake -B build
cmake --build build --config Release

The tools land in build/bin/ (on Windows with Visual Studio, often build\bin\Release\). Add GPU flags such as -DGGML_CUDA=ON to the first command if you also want fast GPU inference for testing; quantizing itself doesn’t need the GPU. Prebuilt binaries from the llama.cpp releases page work too.

Quantize to Q4_K_M and Others

./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M
  1. Input: your 16-bit GGUF.
  2. Output: any file name you like; include the quant type so you can tell files apart.
  3. Type: the quant preset. Names aren’t case-sensitive.
  4. Repeat for each size you want, always starting from the 16-bit file.
Useful option What it does
--imatrix file Use an importance matrix for better quality
--leave-output-tensor Keep output.weight unquantized (bigger, maybe better)
--output-tensor-type, --token-embedding-type Set specific types for those tensors
--pure Disable k-quant mixtures; one type for all tensors
--keep-split Keep the same shards as a split input
--allow-requantize Requantize an already-quantized file (the README warns this can severely reduce quality)

Which Quant Type to Choose

Sizes for Llama 3.1 8B from the llama.cpp README:

Type Bits per weight Size (GiB) Use it when
Q8_0 8.50 7.95 You want near-original quality and have the memory
Q6_K 6.56 6.14 Very high quality, still much smaller
Q5_K_M 5.70 5.33 A step up from Q4 with a little more VRAM
Q4_K_M 4.89 4.58 The usual sweet spot for size and quality
IQ4_XS 4.46 4.17 Slightly smaller than Q4_K_M; best with an imatrix
Q3_K_M 4.00 3.74 Tight on memory
IQ2_M 2.93 2.74 Huge models on small GPUs; needs an imatrix

Qwen’s llama.cpp guide says Q8_0, Q5_K_M and Q4_K_M are common choices for 8B models. A rule of thumb: pick the largest quant that fits fully in your GPU with room left for context. Our RTX 5070 for AI guide shows what fits in 12 GB.

Better Small Quants With an Importance Matrix

An importance matrix tells the quantizer which weights matter most, using sample text. It helps a lot at 4 bits and below, and llama-quantize warns if you make 1-bit or 2-bit quants without one.

./build/bin/llama-imatrix -m my-model-bf16.gguf -f calibration.txt --chunk 512 -o my-model-imatrix.dat -ngl 80
./build/bin/llama-quantize --imatrix my-model-imatrix.dat my-model-bf16.gguf my-model-IQ4_XS.gguf IQ4_XS
  • calibration.txt should be text like your real use: chat, code or your language.
  • -ngl offloads layers to the GPU to speed it up.

Test Your GGUF

  1. Quick chat test:
    ./build/bin/llama-cli -m my-model-Q4_K_M.gguf -p "Explain GGUF in one sentence." -n 64

    Coherent output means the conversion worked.

  2. Compare quality with perplexity on a test file (lower is better):
    ./build/bin/llama-perplexity -m my-model-Q4_K_M.gguf -f wiki.test.raw -ngl 80

    Qwen notes Wikipedia text isn’t a great test for chat models, so also try your own prompts.

  3. Serve it as an API with llama-server: see our llama-server guide.

Run It in Ollama or LM Studio

The No-Setup Option: GGUF-my-repo

The llama.cpp README points to the GGUF-my-repo space on Hugging Face (ggml-org/gguf-my-repo). Sign in, enter a model repo, choose quant types, and it builds the GGUF files and publishes them to your account. It syncs with llama.cpp’s main branch every 6 hours. It’s ideal for public models when you don’t want to install anything.

Troubleshooting

Problem Likely cause Fix
“Model architecture not supported” Old llama.cpp or a brand new architecture git pull, reinstall requirements, retry; otherwise wait for support
Transformers or tokenizer error Model needs a newer transformers pip install -U transformers
Missing tokenizer files Incomplete download Download the full repo, including tokenizer files
Killed or out of memory during conversion Not enough RAM Close apps, add swap, or use GGUF-my-repo
llama-quantize not found Not built, or wrong path Build with CMake; check build/bin or build\bin\Release
Output is gibberish Wrong chat template or broken conversion Test with llama-cli; update llama.cpp and reconvert
Quality much worse than expected Requantized from a quantized file Always quantize from the 16-bit GGUF
Vision model can’t see images No mmproj file Convert with –mmproj and load both files

FAQ

How do I convert a Hugging Face model to GGUF?

Clone llama.cpp, install its requirements, run convert_hf_to_gguf.py on the model folder, then shrink it with llama-quantize.

Which quantization is best?

Q4_K_M is the usual balance of size and quality. Use Q5_K_M, Q6_K or Q8_0 if you have more memory.

Do I need a GPU to convert a model to GGUF?

No. Conversion and quantization run on the CPU. A GPU only speeds up imatrix creation and testing.

Should I use bf16 or f16 when converting?

Use bf16 for most modern models trained in bfloat16, or leave –outtype on auto.

Can I convert a LoRA to GGUF?

Yes, with convert_lora_to_gguf.py in the llama.cpp repo.

Is there an online GGUF converter?

Yes. The GGUF-my-repo space on Hugging Face converts and quantizes public models without any setup.

Can I requantize a Q8 file to Q4?

Only with –allow-requantize, and the README warns quality can drop severely. Start from 16-bit instead.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Laughing woman with a braid in a mustard blazer in a podcast studio holding a card reading GGUF, with the text Make Your Own GGUF