Nemotron 3.5 Bolt: Run NVIDIA’s 74B Model Locally (Ollama, llama.cpp, VRAM, Settings)

NVIDIA’s Nemotron 3.5 Bolt squeezes a 74B mixture-of-experts model into roughly 24 GB. It does this with 2-bit expert weights, quantization-aware distillation and a small built-in drafter for speculative decoding. NVIDIA says it beats Qwen3.6 35B-A3B and Gemma 4 26B-A4B on most agent, browsing and terminal benchmarks.

This guide covers what Bolt is, the exact files and Ollama tags, which GPUs really run it, copy-paste commands for Ollama and llama.cpp, how to switch thinking on and off, and fixes for the common errors.

New to Ollama? My complete Ollama guide covers install, models, commands and fixes in one place.

What Nemotron 3.5 Bolt Is

Bolt is a smaller, compressed child of Nemotron 3.5 Super VL (120B total, 12B active). NVIDIA used its Star Elastic framework to cut the number of experts used per token from 22 to 9. The result is 74B total and 9B active parameters.

The trick that makes it fit in about 24 GB is a mixed quantization plan. NVIDIA trained it to work at these precisions instead of quantizing it afterwards:

NVIDIA calls this quantization-aware distillation (QAD). The 2-bit model is trained to copy the full model’s outputs, which recovers most of the accuracy a plain 2-bit quant would lose.

Bolt also ships with LiLiCorr, a 1.13B drafter model. It guesses several tokens ahead and the big model checks them in one go. In NVIDIA’s vLLM test, on average 4.68 of 7 drafted tokens were accepted with greedy decoding (4.25 with sampling).

The files

Note that the e4b tag in Ollama is a different model. It is an 8B (4B effective) model post-trained by NVIDIA from Gemma 4 E4B, aimed at fast chat and tool use on small machines.

Benchmarks vs Qwen3.6 and Gemma 4

These are NVIDIA’s own figures from the model card. Bolt is tested at its 2-bit 24 GB size, while the rivals are tested at full BF16, which flatters the rivals.

The pattern is clear. Bolt is strongest at agent work: browsing, terminal tasks and multi-step tool use. It is weaker at factual recall (Omniscience), where its extra parameters do not help much at 2 bits.

Independent numbers are lower than NVIDIA’s in places. One write-up notes a gap on HLE between NVIDIA’s figure (28.73) and Artificial Analysis’s (24.7) for the Nemotron 3.5 family. Treat these as maker figures until more people test it.

Hardware: Will It Fit Your GPU?

The “24GB” in the name refers to the size of the model files, not to a 24 GB graphics card. On top of the 22 GB of weights you need room for the vision file, the drafter and the KV cache, which grows with context.

NVIDIA lists Windows and Linux on Ada Lovelace (RTX 40) and Blackwell (RTX 50) GPUs. Older cards may run the GGUF in llama.cpp, but they are not tested, and the NVFP4 Mamba layers run fastest on Blackwell.

Plan for at least 32 GB of system RAM and 30 GB of free disk space. Not sure how the quant names map to quality? Our GGUF quantization guide explains IQ2, Q4 and Q8.

Run It With Ollama

Ollama is the easiest route. Update it first, because Bolt needs the newer Nemotron 3.5 support. Our Ollama guide covers installing it on every OS.

1. Update Ollama to the latest version from ollama.com, or run the installer again.
2. Pull and run the 74B model:

   ollama run nemotron-3.5-bolt

   The latest, 74b-a9b and 74b-a9b-iq2 tags all point to the same 25 GB download.

3. Or the small version:

   ollama run nemotron-3.5-bolt:e4b

4. Check where it loaded:

   ollama ps

   If the PROCESSOR column shows a CPU/GPU split such as “30%/70% CPU/GPU”, part of the model is in system RAM and it will be slower.

Set a sensible context length

The model supports up to 256K tokens, but a huge context eats memory fast. Start with 32K and raise it only if you need to:

ollama run nemotron-3.5-bolt
>>> /set parameter num_ctx 32768

For the API, pass it in options:

curl http://localhost:11434/api/chat -d '{
  "model": "nemotron-3.5-bolt",
  "messages": [{"role": "user", "content": "Write a bash script that backs up ~/projects to a dated zip."}],
  "options": {"num_ctx": 32768, "temperature": 1.0, "top_p": 0.95}
}'

Send an image

Bolt reads images through its C-RADIO vision encoder. In the CLI, drag an image file into the prompt or give its path:

>>> What error is shown in this screenshot? ./error.png

In Python:

from ollama import chat

r = chat(
    model="nemotron-3.5-bolt",
    messages=[{"role": "user", "content": "Describe this chart in 3 bullet points.", "images": ["chart.png"]}],
)
print(r.message.content)

Run It With llama.cpp (Fastest)

llama.cpp gives you full control over the drafter and KV cache, which is how NVIDIA gets its best speeds. Use a recent build, as NVIDIA’s card asks for the latest llama.cpp.

1. Download the three files

pip install -U huggingface_hub
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF --local-dir ./bolt
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF --local-dir ./bolt-draft

You need:

• NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf (main model)
• mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf (vision, optional for text-only use)
• NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf (drafter)

2. Start the server (NVIDIA’s 32 GB GPU recipe)

./build/bin/llama-server \
  --model ./bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --mmproj ./bolt/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --spec-type draft-dflash --spec-draft-n-max 6 \
  --spec-draft-model ./bolt-draft/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf \
  --spec-draft-ngl 999 -ngl 999 \
  --ctx-size 65536 --parallel 1 \
  --batch-size 2048 --ubatch-size 512 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 1.0 --top-p 0.95 --flash-attn on --jinja

On Windows, use llama-server.exe and put each flag on one line, or replace the trailing \ with ^ in Command Prompt.

3. Settings for each memory tier

The 24 GB column is our suggestion based on the file sizes, not a tested NVIDIA recipe. If it still runs out of memory, offload a few layers with -ngl set below the layer count, or use Ollama, which does the split for you.

With two GPUs, add --split-mode layer and, if one card has more free memory, --tensor-split 1,1 adjusted to suit (for example 3,2).

The server exposes an OpenAI-style API at http://localhost:8080/v1, so any OpenAI client can use it. For a deeper look at the tools, see Ollama vs LM Studio vs llama.cpp.

Thinking On, Off and Medium

Bolt can reason before it answers. Reasoning makes it better at hard problems but uses more tokens, and the thinking and the answer share the same max_tokens budget. If answers get cut off, raise max_tokens or turn thinking off.

With llama-server (started with --jinja) or vLLM, pass the switch through the chat template:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")

r = client.chat.completions.create(
    model="bolt",
    messages=[{"role": "user", "content": "Plan a 3-step migration from SQLite to Postgres."}],
    max_tokens=4096,
    temperature=1.0, top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": True, "medium_effort": True}},
)
print(r.choices[0].message.content)

In Ollama, the CLI’s /set nothink and /set think commands, or "think": false in the API, work for models that support thinking. The Ollama library page confirms Bolt can toggle reasoning but does not show an example, so test it on your version.

Using It With Coding Agents

Bolt was trained for agent work, so it suits coding tools. Ollama can launch them directly:

ollama launch claude --model nemotron-3.5-bolt
ollama launch opencode --model nemotron-3.5-bolt

Tips for agent use:

• Give it context room. Agents send long system prompts and tool results. 32K is the minimum that feels usable; 64K is better.
• Use medium thinking. Full thinking on every step slows agents down a lot.
• Watch tool calls. One GitHub issue (ollama/ollama#19057) reported crashes during tool calling caused by a missing kernel. Update Ollama if you hit it.

For hooking local models into MCP tools, see our LM Studio MCP and agent mode guide.

How Fast Is It?

NVIDIA’s own speed test ran llama.cpp on a Surface Laptop Ultra with an RTX Spark chip and 128 GB of memory. On a coding speed benchmark, Bolt with its drafter hit about 84 tokens per second.

So Bolt is not the fastest model in its class. It is slower than Qwen3.6 and Gemma 4, which have fewer active parameters, but about three times faster than the dense Qwen3.8 27B. You trade some speed for better agent scores.

NVIDIA has not published desktop GPU speeds. On a 32 GB RTX 5090 with everything on the GPU, expect it to be faster than the laptop figures. With CPU offload it will be much slower, so keep the whole model on the GPU if you can.

Troubleshooting

FAQ

Can Nemotron 3.5 Bolt run on an RTX 4090?

Probably, with a short context, but NVIDIA does not officially support it. The weights are about 22 GB, which leaves little room for the KV cache on a 24 GB card. Drop the vision file and drafter if needed.

Is Nemotron 3.5 Bolt free for commercial use?

Yes. It uses the OpenMDW License Agreement 1.1, which covers commercial and non-commercial use. Read the licence for the details before shipping a product.

What is the difference between Bolt and Bolt e4b?

They are different models. The default tag is the 74B-A9B MoE model (about 25 GB). e4b is an 8B model built from Gemma 4 E4B (about 7.2 to 7.9 GB) for small GPUs and Macs.

Does Bolt support images and video?

Yes. It takes text, image and video input, and outputs text. In llama.cpp you need the mmproj file for images.

Is it better than Qwen3.6 35B-A3B?

On NVIDIA’s agent and browsing benchmarks, yes. Qwen3.6 is faster, slightly better on PinchBench and fact recall, and needs less memory. Pick Bolt for agents and Qwen3.6 for speed.

Can I fine-tune it?

Use the BF16 checkpoint for fine-tuning. The 24 GB version is already 2-bit and is meant for inference.

Sources

• NVIDIA Research: Nemotron 3.5 Bolt 74B-A9B 24GB (quantization, speed table, repos)
• Hugging Face: Bolt GGUF model card (llama.cpp recipes, benchmarks, reasoning control, licence)
• Ollama library: nemotron-3.5-bolt (tags, sizes, Apple Silicon note)
• SmartScope: Bolt 24 GB GPU compatibility (tag digests, tool-calling issue)