NVFP4 vs MXFP4 in llama.cpp: Which FP4 GGUF to Use on RTX 50, RTX 40 and Mac (2026)

You have probably started seeing NVFP4 and MXFP4 next to the usual Q4_K_M and Q8_0 files on Hugging Face. Both are 4-bit floating-point formats, and llama.cpp now runs both of them.

On an RTX 50-series card, NVFP4 can be much faster than Q4_K_M at reading your prompt. On a Mac, though, NVFP4 does not run in llama.cpp at all.

This guide explains what the two formats are in plain words, which hardware supports each, the llama.cpp builds and commands you need, real benchmark numbers, how they compare with Q4_K_M, where Ollama and LM Studio stand, and fixes for the errors people hit.

FP4 in Plain Words

Classic GGUF quants such as Q4_K_M store each weight as a small integer, plus scales that stretch those integers back to real values. If you want a refresher, my GGUF quantization guide covers the K-quants.

FP4 formats store each weight as a tiny floating-point number instead. The format is called E2M1: 1 sign bit, 2 exponent bits and 1 mantissa bit. That gives you values like 0, 0.5, 1, 1.5, 2, 3, 4 and 6 (positive and negative).

They sit closer together near zero, which is where most weights are.

Four bits on their own are far too coarse, so weights are grouped into blocks that share one scale. The two formats differ in how big those blocks are and how precise the scale is.

The real reason FP4 matters is hardware. NVIDIA Blackwell GPUs (RTX 50 series, RTX PRO Blackwell, B200) have tensor cores that do FP4 maths natively. Integer quants have to be unpacked first, but FP4 can feed the tensor cores much more directly.

NVFP4 vs MXFP4: The Specs

NVFP4’s smaller blocks and finer FP8 scales track the real weights more closely. NVIDIA’s own figures put scale error at 0.08 for E4M3 against 0.72 for E8M0, and report 1% or less accuracy loss against FP8 on DeepSeek-R1-0528. Those are NVIDIA’s numbers, and I have not seen them independently reproduced.

MXFP4’s power-of-two scales are rougher but simpler, and the format is an open standard that more hardware vendors support.

Which Hardware Runs Which

Here is the support picture in llama.cpp, based on source checks dated 3 November 2026:

The key point: NVFP4 runs on older NVIDIA cards, but the big speed-up only comes on Blackwell. On an RTX 3090 or 4090 there are no published NVFP4 speed numbers, so do not assume it beats Q4_K_M there.

If you are on a Mac, read my MLX vs GGUF guide. MLX is usually the faster path on Apple Silicon anyway.

How FP4 Landed in llama.cpp

• MXFP4 arrived first, mainly so gpt-oss could run in its native format. OpenAI post-trained gpt-oss’s MoE weights in MXFP4, so the MXFP4 file is the original, not a re-quant.
• NVFP4 (type 40) landed in spring 2026 through a series of pull requests: #20644 (26 March), then #21074 and #21227 (1 April), with Vulkan work in #21455 and #21539.
• Build b8967 (29 April 2026) added native sm_120 MMQ kernels, which is where the big Blackwell prefill gains come from.
• Fixes are still arriving. The Vulkan scale fix PR #31079 merged on 3 November 2026.

In practice: use a recent build. Any llama.cpp release from the last few weeks has all of the above.

Speed and Size Numbers

Prefill (prompt processing)

Generation (laptop RTX 5090, 24 GB)

One detailed write-up by Bored Consultant ran Qwen3.6-27B at 160K context on a laptop RTX 5090:

Be careful reading this one. The NVFP4 runs also used deeper MTP drafting, so not all of the gain comes from the format. Quality (perplexity) was not measured either.

File sizes

Real NVFP4 GGUFs are not always smaller. Converters often keep attention and the output head in Q8_0 or BF16, so NVIDIA’s Qwen3.6-27B NVFP4 conversion came to 21.5 GB against 17.1 GB for Q4_K_M. Check the actual file size before you assume it fits.

FP4 vs Q4_K_M: Which to Download

On my own rig, two RTX 5080s (Blackwell), NVFP4 is the format I am most interested in for long-context work. I have not published my own benchmarks yet, so everything in this guide is reported numbers. If you are choosing a card, my RTX 5090 vs RTX 5080 for AI comparison covers VRAM trade-offs.

Run an NVFP4 GGUF in llama.cpp

1. Get a recent build. Download the latest CUDA release from the llama.cpp GitHub releases page, or use the ghcr.io/ggml-org/llama.cpp:server-cuda Docker image.
2. Download an NVFP4 GGUF. For example, cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF is popular, with over 3 million downloads in 30 days as of 3 November 2026.
3. Run the server:

llama-server -m Qwen3.8-27B-NVFP4.gguf -ngl 99 -c 32768 -fa on --port 8080

Then open http://localhost:8080 in your browser. For long contexts, quantise the KV cache to save VRAM. Flash attention (-fa on) is required for that:

llama-server -m Qwen3.8-27B-NVFP4.gguf -ngl 99 -c 131072 -fa on --cache-type-k q8_0 --cache-type-v q8_0

In the Bored Consultant test, a 160K-token KV cache took 10.5 GB at f16, 5.6 GB at q8_0 and 2.95 GB at q4_0. That is the difference between fitting a 27B model on 24 GB or not. To benchmark your own card, run:

llama-bench -m Qwen3.8-27B-NVFP4.gguf -ngl 99 -p 512 -n 128

Make Your Own NVFP4 GGUF

This is the part that confuses people: you cannot make NVFP4 with llama-quantize. As of commit b74298c it has no NVFP4 type. The only FP4 option is MXFP4_MOE, which puts MoE expert tensors in MXFP4 and the rest in Q8_0.

Instead, you convert a checkpoint that is already NVFP4, such as an NVIDIA ModelOpt export:

export PYTHONPATH=$PWD/llama.cpp/gguf-py
python llama.cpp/convert_hf_to_gguf.py nvidia/Qwen3.6-27B-NVFP4 --outfile Qwen3.6-27B-NVFP4.gguf --fp8-as-q8

The --fp8-as-q8 flag turns FP8 layers such as attention into Q8_0. NVIDIA’s checkpoints convert cleanly. Some Unsloth “compressed-tensors” checkpoints mix FP8 and NVFP4 groups and fail with “Can’t handle multiple config groups for compressed-tensors yet”. Bored Consultant fixed that with a private patch, but there is no public fix yet.

If you start from a BF16 model, Q4_K_M is still the sensible choice.

Ollama, LM Studio and vLLM

• Ollama: Ollama 0.18 (March 2026) added NVFP4 support in its MLX engine preview on Apple Silicon, which needs 32 GB or more of unified memory. I could not find Ollama release notes confirming NVFP4 GGUF support on CUDA. Ollama uses a vendored copy of llama.cpp that can lag behind, so for NVFP4 on an RTX card, use llama.cpp directly for now.
• LM Studio: it bundles its own llama.cpp runtime, and support follows those runtime updates. I could not confirm a version note for NVFP4, so update the CUDA runtime in LM Studio and test the file. If it fails to load, use llama-server.
• vLLM: supports NVFP4 W4A4 on compute capability 10.0+ and weight-only on older GPUs, according to LLM Compressor docs.

For how the three main tools compare in general, see my Ollama vs LM Studio vs llama.cpp guide.

Troubleshooting

The model fails to load with an unknown tensor type error

Your llama.cpp (or the app’s bundled copy) is too old for NVFP4. Update to a build from after April 2026. The latest release is best.

NVFP4 fails or falls back to CPU on a Mac

Metal does not support NVFP4 matmul. Download an MXFP4, Q4_K_M or MLX version instead.

NVFP4 is no faster than Q4_K_M

On RTX 30 and 40 cards that is expected. On RTX 50, check that you are running a CUDA build (not Vulkan or CPU), with all layers offloaded (-ngl 99). The gain shows mostly in prefill, so test with a long prompt.

Garbled output on Vulkan

Update past the 3 November 2026 scale-lookup fix (PR #31079).

llama-quantize does not list NVFP4

That is correct. Convert an existing NVFP4 checkpoint with convert_hf_to_gguf.py instead.

“Can’t handle multiple config groups for compressed-tensors yet”

The checkpoint mixes FP8 and NVFP4 groups. Use NVIDIA’s ModelOpt checkpoint, or download a ready-made NVFP4 GGUF.

Out of memory with a long context

Quantise the KV cache (--cache-type-k q8_0 --cache-type-v q8_0 with -fa on) or lower -c.

My Take

If you own an RTX 50-series card, NVFP4 is the first quant format in a while that is genuinely worth switching to, especially for long prompts, RAG and coding agents where prefill dominates. On older NVIDIA cards and on Macs, I would stay with Q4_K_M or MLX.

MXFP4 matters mainly because gpt-oss ships in it. For everything else, the choice is really NVFP4 on Blackwell or Q4_K_M everywhere else.

FAQ

What is the difference between NVFP4 and MXFP4?

Both use 4-bit E2M1 weights. NVFP4 uses blocks of 16 with FP8 scales (4.5 bits per weight). MXFP4 uses blocks of 32 with power-of-two scales (4.25 bits per weight). NVFP4 is more accurate, and MXFP4 is a more widely supported open standard.

Does llama.cpp support NVFP4?

Yes. It has been there since spring 2026, with native Blackwell kernels from build b8967 (29 April 2026). It runs on CUDA, Vulkan and CPU, but not on Apple Metal.

Is NVFP4 better than Q4_K_M?

On RTX 50 cards, reported prefill is 24-40% faster. On older cards there is no proven gain. Quality is close, but file sizes vary by converter.

Can I run NVFP4 on an RTX 4090?

Yes, through normal CUDA kernels, but there are no published speed-ups. Q4_K_M is a safe choice there.

Does Ollama support NVFP4?

Ollama’s MLX engine preview on Apple Silicon added NVFP4 in version 0.18. I could not confirm NVFP4 GGUF support on CUDA, so use llama.cpp directly on RTX cards.

How do I make an NVFP4 GGUF?

Convert an existing NVFP4 checkpoint (such as an NVIDIA ModelOpt export) with convert_hf_to_gguf.py. llama-quantize cannot create NVFP4.

Why is gpt-oss in MXFP4?

OpenAI post-trained gpt-oss’s MoE weights in MXFP4, so that is its native format, not a later quantisation.

Does NVFP4 work on AMD?

llama.cpp has ROCm kernel settings for RDNA2 to RDNA4, but there is little public testing. Treat it as experimental.

Sources

• InsiderLLM: FP4 in llama.cpp, NVFP4 vs MXFP4 (PRs, builds, RTX 5090 prefill)
• OpenClaw DC: NVFP4 vs MXFP4 (source checks 3 Nov 2026)
• Bored Consultant: Qwen3.6-27B NVFP4 on llama.cpp (laptop RTX 5090)
• Summary of the Ollama blog: MLX on Apple Silicon preview (NVFP4 in 0.18)
• GitHub: ggml-org/llama.cpp