Best Way to Run Recursive Flint Locally on a 16 GB GPU (NVFP4 + MoE Offload Guide)

Recursive’s Flint is the open model everyone in local AI has been talking about this week. It’s a 40B mixture-of-experts coding model that only activates about 2.5B parameters per token, it’s Apache 2.0, and Recursive pitches it as a laptop-class coding agent. The catch for people like me with a 16 GB card (I’m on an RTX 5080) is that none of the 4-bit files fit in VRAM: the official NVFP4 GGUF is 22.9 GB. This guide is the setup I’d use to run Flint locally on a 16 GB GPU anyway: which file to pick, the llama.cpp flags that keep the right weights on the GPU, the arithmetic behind those flags, and the settings Recursive recommends so the model behaves the way it did in their evaluations.

One thing up front: I haven’t benchmarked Flint on my own RTX 5080 yet, so I’m not going to give you made-up tokens-per-second numbers. Every number below is either from Recursive, Unsloth, llama.cpp or a third-party test I link to, or my own arithmetic from Flint’s published config.json, and I say which is which.

What Recursive Flint Is (and Why It’s Trending)

Flint is the first model from Recursive, Richard Socher’s self-improving-AI startup. According to Recursive’s launch post, a small team plus its automated research system (the “Eureka Machine”) built it from scratch in under four months, and the Eureka Machine helped with everything from the training recipe to quantization and inference kernels. TechCrunch covered the launch on October 21, and on Hugging Face RecursiveSI/flint and the community GGUFs have been near the top of the trending list all week.

The specs from the official model card:

Property	Flint
Total / active parameters	39.8B / ~2.5B per token (including the output head)
Layers	49: 2 dense, then 47 MoE layers
Experts	256 routed per MoE layer, 6 selected per token, plus 1 always-on shared expert
Attention	35 sliding-window layers (513-token window) and 14 full-attention layers; 32 query heads, 4 KV heads, head dim 128
Context	262,144 tokens (prompt plus output)
Input / output	Text / text, with reasoning and multi-turn tool calls
License	Apache 2.0

Recursive’s own scores for the BF16 model are 74.2 on SWE-bench Verified, 54.9 on Terminal-Bench 2.1 and 80.3 on LiveCodeBench v6. Treat those as vendor numbers; I come back to independent results in the last section.

Why a 40B Model Can Work on a 16 GB GPU

This is the part most “run it locally” posts skip, and it’s the reason the flags later on make sense. I added up Flint’s weights from the numbers in its config.json (hidden size 2048, expert width 512, shared expert width 1024, 256 experts, 47 MoE layers):

• Routed experts: 256 experts × 3 matrices × 2048 × 512 = about 805M parameters per MoE layer, so about 37.9B across 47 layers. That’s roughly 95% of the model.
• Everything else (attention in all 49 layers, the shared experts, the two dense layers, routers, embeddings and output head): about 1.9B.

Those two figures add up to the 39.8B on the model card, which is a good sign the math is right. Now the key point: per token, each MoE layer only uses 6 of its 256 experts. So if an expert layer sits in system RAM, the CPU only has to read about 2.3% of that layer’s weights for each token, not the whole layer. That’s why offloading expert weights to the CPU costs far less speed than offloading whole layers of a dense model, and why Flint is a much better fit for a 16 GB card than a dense 27B model.

The practical rule that falls out of this: always keep attention, the shared expert and the KV cache on the GPU, and move expert tensors to the CPU first. llama.cpp has a flag that does exactly that.

Which Flint File to Download

File sizes below come straight from the Hugging Face repos and the Ollama tag list.

File	Size	Fits fully in 16 GB?	My take
RecursiveSI flint-nvfp4.gguf	22.9 GB	No, needs expert offload	First choice on RTX 50-series. It’s the official quant, made with Recursive’s distillation recipe, and NVFP4 is the 4-bit format Blackwell tensor cores support natively.
Unsloth UD-Q4_K_XL	25.1 GB	No	The 4-bit K-quant for RTX 30/40 cards. Needs more offload than NVFP4.
Unsloth UD-Q3_K_XL	18.5 GB	No, but close	Unsloth’s suggested pick if memory is tight. Needs the least offload of the “real” quants.
Unsloth UD-IQ2_XXS / UD-IQ1_M	11.9 / 11.4 GB	Yes	Fits with room for context, but I’d only use it if you have little system RAM. Flint is quantization-sensitive (see gotchas).
RecursiveSI flint-int4-q4_1.gguf	26.0 GB	No	Recursive says this one is “primarily intended for Apple Silicon”. Skip it on NVIDIA.
Ollama flint:40b	23 GB	No	Easiest install; less control over the GPU/CPU split.

Why I prefer the NVFP4 file on a 50-series card: Recursive’s launch post explains that its quantization search quantized even the output layer to 4 bits and then ran a distillation stage with the 16-bit model as teacher, and that the final NVFP4 model scores within 0.7 points of 16-bit across its benchmarks. The model card’s own table shows NVFP4 at 74.1 vs 74.2 on SWE-bench Verified. The NVFP4 GGUF README also lists RTX PRO 6000 Blackwell and RTX Spark among the tested NVIDIA hardware. A generic K-quant doesn’t get that kind of model-specific tuning.

Step by Step: Run Flint in llama.cpp on Windows

Get a Flint-Capable llama.cpp Build

Flint support was merged into llama.cpp in PR #30928 on October 22, 2026 (IST). Atomic Agent’s write-up pins that to build 11803, and notes that the chat template inside the GGUFs needs the string-escape fix from build 11798. Anything newer is fine; the newest release when I wrote this was b11871.

1. Open the llama.cpp releases page.
2. For an RTX 50-series card on Windows, download llama-bXXXXX-bin-win-cuda-13.4-x64.zip and the matching cudart-llama-bin-win-cuda-13.4-x64.zip. Blackwell needs a recent CUDA, so I’d use the 13.x package rather than 12.4.
3. Extract both zips into the same folder, for example C:\llama, and run .\llama-server.exe --version to confirm the build number.

Download the Model

Install the Hugging Face CLI (pip install -U huggingface_hub), then:

hf download RecursiveSI/flint-nvfp4-gguf flint-nvfp4.gguf opencode.json pi-models.json --local-dir C:\models\flint-nvfp4-gguf

The two JSON files are ready-made client configs for OpenCode and pi; they’re tiny and worth grabbing. For an RTX 30/40 card, swap in hf download unsloth/flint-GGUF flint-UD-Q3_K_XL.gguf --local-dir C:\models\flint.

Launch llama-server with Expert Offload

Recursive’s official command in the NVFP4 README assumes 48 GB of GPU memory and the full 262K context. Here’s the same command adapted for 16 GB, in PowerShell:

.\llama-server.exe -m C:\models\flint-nvfp4-gguf\flint-nvfp4.gguf `
  --alias flint -ngl all --n-cpu-moe 25 `
  -c 32768 -np 1 -fa on -ctk f16 -ctv f16 --jinja `
  --temp 1 --top-p 0.95 --top-k 20 --min-p 0 `
  --host 127.0.0.1 --port 8080

What each change does and why:

• -ngl all puts every layer on the GPU first, then --n-cpu-moe 25 overrides that for the expert weights only of the first 25 layers, which stay in system RAM. Attention, the shared experts and the KV cache for every layer remain on the GPU.
• -c 32768 instead of 262144. On a 16 GB card, VRAM spent on context is VRAM that can’t hold experts. Start at 32K and raise it once you know your headroom.
• -np 1 forces a single server slot. llama-server’s default is “auto”, and extra slots mean extra KV cache you don’t need for a personal coding agent.
• --port 8080 because current llama-server builds default to port 9931, while most agent configs (including Recursive’s) expect 8080.
• Sampling flags match Recursive’s recommendations, so clients that don’t send their own sampling settings still get the right ones.

Once it’s up, curl http://127.0.0.1:8080/health should answer, and the web UI is at http://127.0.0.1:8080.

Tune –n-cpu-moe for Your Card

The 25 above isn’t a measured value. It’s a starting point from this arithmetic, and you should tune it:

• One MoE layer’s experts are ~805M parameters. NVFP4 stores about 4.5 bits per weight (4-bit values plus a scale per 16 values), so each layer moved to the CPU frees roughly 0.45 GB of VRAM.
• The NVFP4 file is 22.9 GB. On a 16 GB card running a Windows desktop I’d aim for about 12.5 GB of weights on the GPU, leaving room for a 32K KV cache, compute buffers and the desktop. That means moving ~10.4 GB, or about 23 expert layers.
• --n-cpu-moe N counts from layer 0, and layers 0 and 1 are dense (no experts). So 23 expert layers needs N = 25.

The same math gives starting points of about N = 18 for UD-Q3_K_XL (~0.37 GB per layer) and N = 27 for UD-Q4_K_XL (~0.51 GB per layer). Then:

1. Start the server and watch dedicated GPU memory in Task Manager or nvidia-smi while you send a long prompt.
2. If you have more than ~1 GB free, lower N by 2 and restart. Each step moves another chunk of experts back to the fast side.
3. If you get a CUDA out-of-memory error, or Windows starts spilling into shared GPU memory (speed falls off a cliff), raise N.

Recent llama.cpp builds also have --fit, which is on by default and adjusts unset arguments to fit device memory. If you leave out -ngl and --n-cpu-moe entirely, it’ll try to pick a split for you. That’s a reasonable first run; read the load log to see what it chose, then pin your own values once you know what works. Make sure you have enough system RAM for the offloaded part: at N = 25 that’s roughly 10 GB of expert weights on top of Windows and your editor, so 32 GB of RAM is comfortable and 16 GB is tight.

Context Length and KV Cache Math

Flint’s attention design is very friendly to long context. Only the 14 full-attention layers keep a cache that grows with context; the 35 sliding-window layers only cache their 513-token window, as long as you leave --swa-full unset (Recursive’s README says this explicitly). From the config (4 KV heads × 128 dims × K and V × 2 bytes for f16), each token costs about 28 KB across the 14 full layers. My rough estimates for the growing part of the cache, f16, one slot:

Context	Full-attention KV (f16, approx.)
32,768	~0.9 GB
65,536	~1.9 GB
131,072	~3.8 GB
262,144	~7.5 GB

These are lower bounds from the architecture; llama.cpp adds compute buffers and the small sliding-window cache on top. The trade-off on 16 GB is direct: each extra 32K tokens of context costs about the same VRAM as two expert layers. For agentic coding on a mid-sized repo I’d go 32K to 64K and raise --n-cpu-moe by 2 when I move to 64K. Don’t drop the KV cache to q4_0 to squeeze more in; see the gotchas below.

The Easy Routes: Ollama, LM Studio and Unsloth

Ollama. Flint is in the official library: ollama run flint pulls flint:40b, a 23 GB model with a 256K context window. Ollama splits the model between GPU and CPU on its own, so it runs on 16 GB, but you don’t get llama.cpp’s expert-level control. Ollama also doesn’t load the full 256K context by default; set the context you want with /set parameter num_ctx 32768 in the chat, or the OLLAMA_CONTEXT_LENGTH environment variable for the server. The library page lists one-line launchers for agents, for example ollama launch opencode --model flint and ollama launch claude --model flint.

LM Studio. The lmstudio-community/flint-GGUF repo is there, and Recursive lists LM Studio as supported. Make sure the bundled llama.cpp runtime is updated to a Flint-capable version. If your version has the option to force MoE expert weights onto the CPU, turn it on and push GPU offload to the max; that’s the same idea as --n-cpu-moe.

Unsloth. Unsloth’s Flint guide covers Unsloth Desktop and an unsloth run --model unsloth/flint-GGUF:UD-Q4_K_XL server with the recommended sampling flags. Its hardware table says 16 to 20 GB of combined RAM + VRAM for 3-bit and 26 GB for 4-bit.

Connect a Coding Agent

Flint is built for agent loops, not just chat, so the payoff is wiring it into a coding agent. With llama-server running on port 8080, Recursive’s NVFP4 repo ships configs for two agents:

# OpenCode
$env:OPENCODE_CONFIG = "C:\models\flint-nvfp4-gguf\opencode.json"
opencode --standalone

# pi
mkdir .pi-flint
copy C:\models\flint-nvfp4-gguf\pi-models.json .pi-flint\models.json
$env:PI_CODING_AGENT_DIR = "$PWD\.pi-flint"
pi --provider flint-local --model flint

One thing to fix before you start: those configs assume 262,144 tokens of context and a 65,536-token generation budget. The README says to match the client’s context limit to your -c value, so edit the JSON to 32768 (or whatever you chose). Otherwise the agent will happily build prompts larger than the server can hold. Any other OpenAI-compatible client works too: point it at http://127.0.0.1:8080/v1 with model name flint.

Recommended Settings and Gotchas

• Sampling: temperature 1.0, top-p 0.95, top-k 20, min-p 0. That’s what Recursive used for every evaluation. Many agents default to temperature 0.2; Atomic Agent ran its test that way, so results may differ.
• Keep the reasoning: Flint always reasons before answering. The model card says to preserve the returned reasoning_content in conversation history; clients that strip it make multi-turn tool use worse.
• Output budget: at least 32,768 tokens for hard problems, reasoning included. Prompt and output share the context, which is one more reason not to run a tiny -c.
• Don’t over-compress: Atomic Agent cites community tests showing Flint’s top-1 token agreement around 85 to 88% at roughly 4 bits per weight, and q4_0 KV cache raising divergence on code. Keep the KV cache at f16 or q8_0, and treat 2-bit files as a last resort.
• Expect llama.cpp to get faster: the Flint PR uses an adaptive expert-activation scaling step, and its author notes it slows inference by 10 to 30% for now, to be fixed in follow-up PRs. Update your build every week or so.
• Break tasks down: Recursive itself says Flint “may need more guidance” than big closed models and recommends splitting complex work into smaller subtasks with explicit instructions.

How Good Is Flint, Honestly?

Vendor benchmarks put Flint at the top of MoE models under 50B for agentic coding, but they also show the dense Qwen 3.8 27B clearly ahead on Terminal-Bench 2.1 (73.0 vs 54.9) and LiveCodeBench v6 (90.3 vs 80.3). Flint’s argument is speed per quality, not raw quality.

The most useful independent test I’ve found is Atomic Agent’s, on a rented RTX 5090 with llama.cpp build 11825 and the Unsloth UD-Q4_K_XL file. Flint passed 12 of 18 agent runs, decoded at 145.7 tokens per second, and used 27.4 GB of VRAM at a 131K context. It also failed a log-counting task in all three runs by writing a number without actually counting, which is a good reminder to verify what any local agent says it did. Remember that a 5090 holds the whole model in VRAM; with expert offload on a 16 GB card, expect lower speed.

My take: for a 16 GB card owner who wants a private, free coding model that’s fast enough to keep an agent loop moving, Flint with the official NVFP4 file and partial expert offload is the setup to try this week. If you mostly want maximum quality and don’t care about speed, a dense model like Qwen 3.8 27B is still the stronger coder. If you’re curious about the bigger open releases from the same week, see my posts on Kimi K3.1 Pro and Muse Spark Open, neither of which is realistic on a single 16 GB card.

FAQ

Can Recursive Flint run on a 16 GB GPU?

Yes, with partial offload. No 4-bit Flint file fits in 16 GB (the official NVFP4 GGUF is 22.9 GB), but because Flint only uses 6 of 256 experts per token, you can keep attention and the shared expert on the GPU and move some expert layers to system RAM with llama.cpp’s --n-cpu-moe. The 2-bit Unsloth files (about 11.4 to 11.9 GB) fit fully but lose more quality.

Which Flint quant should I use on an RTX 5080 or 5090?

The official RecursiveSI/flint-nvfp4-gguf file. Recursive reports the NVFP4 build within 0.7 points of BF16, and NVFP4 is native to Blackwell. A 5090 (32 GB) can hold it fully; a 5080 or 5070 Ti (16 GB) needs expert offload.

Which llama.cpp version supports Flint?

Build 11803 or newer, where PR #30928 added the FlintForCausalLM architecture. Older builds can’t load the model, and builds before 11798 can’t parse the chat template embedded in the GGUFs.

Should I use Ollama or llama.cpp for Flint?

Ollama (ollama run flint) is the quickest way to try it. On a 16 GB card I’d move to llama.cpp once you’re serious, because --n-cpu-moe lets you decide exactly which weights stay in VRAM, and you can match Recursive’s recommended server settings.

Is Flint free to use?

Yes. The weights are Apache 2.0, so running them locally costs nothing beyond your hardware. Atomic Agent also reported a free, rate-limited OpenRouter listing (recursive/flint-v1.0:free) as of October 22, 2026; free API listings can change at any time.

Does Flint understand images?

No. Flint is text in, text out. It’s meant for code: reading repos, fixing bugs, implementing features and writing tests through tool calls.

What context length should I use locally?

On 16 GB, 32K to 64K. Flint supports 262,144 tokens, but each 32K tokens of f16 cache costs roughly 0.9 GB of VRAM that could otherwise hold expert weights. Recursive recommends 48 GB or more of GPU memory for the full context.