Nemotron 3.5 Bolt E4B 4GB: Which GGUF to Download, Real VRAM Use and RTX 5080 Speed Test

Short answer: for NVIDIA’s new Nemotron 3.5 Bolt E4B 4GB, download the official Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf (6.18 GB) and run it in llama.cpp with every layer on the GPU. Despite the 6.18 GB file, my RTX 5080 only needed about 3.5 GB of VRAM at 8K context and 4.4 GB at the full 64K, and it generated text at about 145 tokens per second. The optional DFlash drafter (0.74 GB) was worth it for reasoning and math, where it doubled speed. For very short chat replies it made things slightly slower.

What Is Nemotron 3.5 Bolt E4B 4GB?

Nemotron 3.5 Bolt E4B 4GB is a small language model that NVIDIA built on top of Google’s Gemma-4-E4B-it and then post-trained with its own Nemotron recipe. According to the official model card, it has 8B total parameters (4B effective), takes text in and returns text out, handles up to 64K tokens of context, and went GA on Hugging Face and build.nvidia.com on October 23, 2026. NVIDIA used reinforcement learning (GRPO) to train it for multi-turn tool use, instruction following, structured output and staying in character.

The “4GB” in the name is NVIDIA’s memory target, not the file size. The launch blog says the model is designed to fit “into as little as ~4GB in its target workloads,” which leaves VRAM for a game, a speech model or an image model in the same pipeline. That is exactly why it interests me. On my PC, the GPU is usually shared with ComfyUI and a TTS server. A chat or agent model that stays under 5 GB can sit next to them without anything being evicted.

NVIDIA’s benchmark table compares it with Gemma 4 E4B, Qwen 3.5 4B and its own older Nemotron 3 Nano 4B. Bolt wins most of the instruction-following and persona tests. For example, IHEval is 84.8 vs 67.0 for Gemma 4 E4B, and RoleBench is 67.0 vs 49.9. Qwen 3.5 4B stays ahead on Tau2-Bench (75.1 vs 55.1) and on long-context RULER. These are NVIDIA’s own numbers from its NeMo Gym harness, not my tests. The takeaway is that Bolt is tuned to be a fast, obedient assistant, not a coding or long-document model.

Which Nemotron 3.5 Bolt E4B GGUF File Should You Download?

NVIDIA ships only one quantization itself, and it’s the one I recommend. Here are the files I found on Hugging Face as of Oct 26, 2026:

File	Repo	Size	Use it when
Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf	nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB	6.18 GB	Default choice. NVIDIA calibrated this Q4_K_M with an importance matrix built from about 1M tokens of its own RL rollouts, and its published benchmarks use this exact file.
Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf	nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash	0.74 GB	Optional speculative-decoding drafter. Add it if your prompts involve reasoning, math or structured output.
NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q4_K_XL.gguf … UD-Q8_K_XL.gguf	unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF	5.06 GB to 8.19 GB	If you want a higher-bit file (Q6/Q8) and have VRAM to spare. Unsloth also ships mmproj files.
NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.gguf	unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF	15.05 GB	Only for re-quantizing or reference testing. It isn’t needed on a 16 GB card.

Why do I prefer NVIDIA’s own Q4_K_M over a bigger community quant? Because NVIDIA’s benchmark table includes a separate Q4_K_M column, and the scores barely move from BF16. On several rows they even go up: MultiChallenge is 55.6 vs 53.1, and RoleBench is 71.2 vs 67.1. With a quant that already matches the full-precision model on the vendor’s own evals, a Q8 file mostly costs you VRAM and speed. Put both GGUFs in one folder, for example C:\llm\bolt\. llama.cpp doesn’t care where they live, as long as you pass the full paths.

Why Is the “4GB” Model a 6.18 GB File?

This was the first thing I wanted to check, because “nemotron 3.5 bolt e4b 4gb” is what people are searching for and the download is clearly larger than 4 GB. llama-bench reports the file as gemma4 E4B Q4_K - Medium with 7.52B parameters. The “E” in E4B stands for effective. Gemma’s E-models store a large share of their weights as per-layer embedding tables, which are lookups rather than heavy matrix maths. llama.cpp keeps embedding lookups like these in system RAM instead of VRAM, and my measurements below match that.

I measured the result with nvidia-smi before and after loading the model on an otherwise idle RTX 5080:

• 8K context: +3,593 MiB (about 3.5 GB) of VRAM.
• 64K context (the maximum): +4,545 MiB right after load, and +4,659 MiB after a 12.6K-token prompt.
• 64K plus the DFlash drafter: +6,315 MiB after load, and +6,543 MiB after the same tests.

So the name is honest about VRAM. Just budget around 6 to 7 GB of free system RAM and disk space for the file, and keep the context at 8K to 16K if you’re squeezing it onto a 6 GB or 8 GB laptop GPU. Unsloth’s guide gives the same guidance: “~4GB VRAM + system RAM” for this file.

How Do You Run Nemotron 3.5 Bolt E4B in llama.cpp on Windows?

NVIDIA’s quick start uses a Linux Docker image (ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459), so any llama.cpp build from b11459 onward should load the model. On Windows I skipped Docker and used the prebuilt release, which takes about five minutes:

1. From the llama.cpp releases page, download llama-bXXXXX-bin-win-cuda-13.4-x64.zip plus the matching cudart-llama-bin-win-cuda-13.4-x64.zip. I used b11889, released on Oct 26, 2026. Extract both zips into the same folder.
2. Download the two GGUF files from the NVIDIA repos linked above. In a browser, use the “download” icon next to each filename. If you have the CLI, run hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf --local-dir C:\llm\bolt.
3. Start the server. This is the plain version I benchmarked:

llama-server.exe -m C:\llm\bolt\Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf ^
  --alias nemotron-bolt --host 127.0.0.1 --port 8000 ^
  -c 65536 -ngl 999 --flash-attn on --jinja

Add these flags to turn on the DFlash drafter. They’re NVIDIA’s values from the model card:

  --spec-draft-model C:\llm\bolt\Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf ^
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-ngl 999

A few notes on the flags. --jinja is required, because the thinking and conversational switches live in the chat template. -ngl 999 puts every layer on the GPU. -c 65536 is the full context, and you can lower it to 8192 to save about 1 GB of VRAM. If you have two GPUs, as I do, set CUDA_VISIBLE_DEVICES=1 (with CUDA_DEVICE_ORDER=PCI_BUS_ID) to pin the model to one card. The server exposes an OpenAI-compatible API at http://127.0.0.1:8000/v1, so any OpenAI client works. NVIDIA recommends temperature 1.0 and top_p 0.95 in both thinking modes, and I used those settings for every test below.

If you’ve followed my Recursive Flint 16 GB guide, this is far simpler. There’s no MoE offload and no tensor overrides, because the whole model fits comfortably.

My RTX 5080 16 GB Test Results

Test setup: RTX 5080 16 GB (the second card in my PC, otherwise idle), Intel i7-14700K, Windows 11, llama.cpp b11889 CUDA 13.4 build, Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, flash attention on and f16 KV cache. Every chat prompt ran three times at temperature 1.0 and top_p 0.95, and I report the average of the server’s own timings output.

Test	Without drafter	With DFlash drafter
llama-bench prompt processing (pp512 / pp4096)	8,492 avg (9,360 after warm-up) / 9,303 tok/s	not tested
llama-bench generation (tg128)	163.2 tok/s	not tested
~300-word explanation, thinking off (avg 280 tokens)	145.1 tok/s, ~2.0 s	165.8 tok/s, ~1.9 s
Same explanation, thinking on (avg ~470 tokens)	146.3 tok/s, ~3.3 s	182.2 tok/s, ~2.7 s
Math word problem, thinking on (~150 tokens)	146.4 tok/s, ~1.2 s	330.1 tok/s, ~0.5 s
Conversational mode, one-line reply (~14 tokens)	139.7 tok/s, 0.16 s	118.1 tok/s, 0.18 s
12,620-token prompt (time to first token)	1.45 s (8,714 tok/s)	2.61 s (4,841 tok/s)
VRAM added at 64K context	4,545 MiB	6,315 MiB
Server load time (file already cached)	2.6 s	3.6 s

For a 4B-effective model, that’s very fast. A full 300-word answer arrives in about two seconds, and a short assistant reply comes back in under a fifth of a second. That’s the latency you need for a voice pipeline, where speech-to-text and TTS also take their share. NVIDIA’s blog lists 106 to 133 tok/s on an RTX 5070, depending on context length. My 145 tok/s on a 5080 is in line with that.

Here are real outputs from those runs, unedited apart from shortening.

Math word problem (thinking on). The prompt was “A shop sells pens at 3 for 40 rupees. How much do 27 pens cost?” The final answer was simply 360 rupees. The reasoning trace was short and correct: “How many groups of 3 are in 27? 27 ÷ 3 = 9 … 9 × 40 = 360.” The whole trace plus answer was 145 to 188 tokens, which fits NVIDIA’s claim that Bolt is tuned to produce fewer tokens.

Conversational mode. The prompt was “Hey, I’m setting up my new gaming PC tonight. Any quick tips?” The reply was: “Cable management, good airflow, and don’t over-volt.” That’s one sentence with no Markdown, which is exactly what you want to feed a TTS voice. In normal mode the same model writes headed, bolded paragraphs.

Tool call. I gave it a get_weather tool and asked “What’s the weather in Ludhiana right now?” It returned finish_reason: tool_calls with get_weather({"city":"Ludhiana"}) in 0.19 s, with no extra text and the correct arguments. It did the same with the drafter on.

Is the DFlash Drafter Worth It?

DFlash is NVIDIA’s parallel drafter. It proposes up to 7 tokens in a single forward pass, and the main model verifies them all at once. Because the target model still checks every token, the output distribution is unchanged. NVIDIA says it gives up to 21% higher throughput than Gemma 4 E4B’s original MTP drafter. My results show the gain depends heavily on what you ask, because it comes down to how many drafted tokens are accepted. llama.cpp prints this as “draft acceptance”:

• Math and step-by-step reasoning: 55% to 63% acceptance and a mean accepted length of about 5 tokens, for a 2.25x speedup (146 to 330 tok/s). Predictable, formulaic text is ideal for drafting.
• Free-form explanations: 19% to 26% acceptance, for 14% to 25% faster generation.
• Tiny conversational replies: 7% to 17% acceptance. Rejected drafts are wasted work, so the replies were about 15% slower. They still finished in under 0.2 s, so you won’t notice.
• Long prompts: prompt processing dropped from 8,714 to 4,841 tok/s, because the drafter has to read the prompt too. A 12.6K-token prompt took 2.6 s to the first token instead of 1.45 s.

My rule: turn DFlash on for an agent that reasons, does maths or emits JSON and tool calls in thinking mode. Leave it off for a voice assistant in conversational mode, for RAG with very long prompts, or when you need the extra ~1.7 GB of VRAM for something else.

How Do Thinking, Conversational Mode and Tool Calling Work?

All three switches go through the chat template, which is why --jinja matters. With the OpenAI client, pass them in extra_body:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="nemotron-bolt",
    messages=[{"role": "user", "content": "Any quick tips for a new PC?"}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
    extra_body={"chat_template_kwargs": {
        "enable_thinking": False,      # True is the default
        "conversational_mode": True     # short, speech-ready replies
    }})
print(r.choices[0].message.content)

• Thinking (enable_thinking): on by default. llama-server returns the trace separately in reasoning_content, so your app can hide it. In my explanation test, thinking added only about 190 tokens and roughly 1.3 s. To cap it, start the server with --reasoning on --reasoning-budget N, as NVIDIA’s card shows.
• Conversational mode: concise replies with little formatting, made for speech. You can combine it with thinking and with tools.
• Tool calling: pass a standard OpenAI tools array. One catch from the model card: the template rejects “stray” tool messages that don’t directly follow an assistant tool call. For slow tools, NVIDIA suggests returning a “pending” tool result right away and sending the real result in a later turn.

If you’re building a local voice assistant, pair this with a local TTS. My Voicebox tutorial covers the voice side on Windows. Bolt plus a TTS model still leaves most of a 16 GB card free.

Bolt E4B vs Bolt 74B-A9B: Which One Fits a 16 GB GPU?

NVIDIA released two Bolt models, and search results mix them up:

	Bolt E4B 4GB	Bolt 74B-A9B 24GB
Parameters	8B total, 4B effective (Gemma 4 E4B)	74B total, 9B active (Mamba-2/Transformer hybrid MoE)
Official GGUF	Q4_K_M, 6.18 GB	2-bit QAT, 22.02 GB plus a 1.73 GB mmproj
Inputs	Text	Text, images, video
Context	64K	Up to 256K
NVIDIA’s deployment target	~4 GB VRAM, from Jetson up to RTX	24 GB+ memory; card recipes for 28 GB and 32 GB
On my 16 GB RTX 5080	Fits fully, 3.5 to 4.4 GB, tested	Doesn’t fit on the GPU alone, not tested

On a 16 GB card, the 74B-A9B can only run by spilling part of the model into system RAM, which costs a lot of speed. There are community “16GB” REAP-pruned 2-bit variants (around 13.4 GB), but those remove experts from an already 2-bit model. I haven’t tested them, so I wouldn’t use them for anything that matters. If you want a bigger brain on 16 GB, a mid-size MoE with expert offload is the better route, as I showed in the Flint guide. If you want an image-aware model that fits on the card, see MiniCPM-V 4.7 on a 16 GB GPU. For a fast assistant that leaves room for ComfyUI, E4B is the right pick. My ComfyUI VRAM test shows how much the image and video side needs.

Can You Use Ollama or Unsloth Instead?

Yes. Ollama’s library has a nemotron-3.5-bolt page where the e4b and e4b-q4_K_M tags were added around Oct 24, 2026. Note that nemotron-3.5-bolt:latest is the 25 GB 74B model, so type ollama run nemotron-3.5-bolt:e4b explicitly. Ollama lists those E4B tags with a 128K context and image input, while NVIDIA’s card says 64K and text-only. I’d go with NVIDIA’s limits until that’s cleared up. Unsloth’s desktop app and its Nemotron 3.5 Bolt guide are another route. Both are easier than llama.cpp. I only benchmarked llama.cpp, though, and all the DFlash numbers above come from the llama-server flags shown earlier.

FAQ

How much VRAM does Nemotron 3.5 Bolt E4B need?

In my test, the official Q4_K_M used about 3.5 GB of VRAM at 8K context and 4.4 GB at the full 64K context in llama.cpp. Adding the DFlash drafter raised the 64K figure to about 6.2 GB. Any 6 GB or larger NVIDIA GPU can run it, and an 8 GB card has room to spare.

Why is the Nemotron 3.5 Bolt E4B 4GB download 6.18 GB?

The file holds all 7.5B stored parameters, including large per-layer embedding tables from the Gemma 4 E4B architecture. Those don’t need to sit in VRAM, so the GPU footprint stays around 4 GB while the rest lives in system RAM.

How fast is Nemotron 3.5 Bolt E4B on an RTX 5080?

About 163 tok/s in llama-bench and around 145 tok/s in real chat requests, with roughly 9,300 tok/s of prompt processing. With the DFlash drafter, a math prompt reached 330 tok/s.

Can I use Nemotron 3.5 Bolt E4B commercially?

NVIDIA’s model card says it’s ready for commercial and non-commercial use under the OpenMDW License 1.1. Because it’s post-trained from Gemma 4, the Gemma 4 base is also covered by Apache 2.0, as the card notes. Read both licenses before you ship a product.

Does Nemotron 3.5 Bolt E4B understand images?

NVIDIA’s E4B model card lists text input and text output only. For images and video, NVIDIA points to the larger Bolt 74B-A9B, which has its own vision projector.

Should I enable the DFlash drafter?

Enable it for reasoning, maths and structured or tool-heavy output, where I saw 1.25x to 2.25x faster generation. Skip it for short conversational replies and very long prompts, where it was slower in my tests, or if you’re short on VRAM.