How to Run Qwen3.8-Flash-Next Locally (2026): 2x 16 GB GPUs + 128 GB RAM llama.cpp Guide

Qwen3.8-Flash-Next is the most interesting open model of late 2026 for home users: a 125B mixture-of-experts model that only activates about 6B parameters per token, scores close to frontier models on coding benchmarks, sees images and video, and still runs on a normal desktop if you have enough system RAM. It is also a model I run on my own PC, which has two RTX 5080 16 GB cards and 128 GB of RAM, so this guide is built around that kind of setup. Below I show you exactly how to run Qwen3.8-Flash-Next locally with llama.cpp on a 32 GB VRAM + 128 GB RAM machine: which GGUF to pick, the full download and launch commands for Windows and Linux, the right settings for thinking and non-thinking mode, vision, the optional MTP speed-up, Ollama and LM Studio routes, and fixes for the errors people actually hit.

What Is Qwen3.8-Flash-Next?

Alibaba’s Qwen team released Qwen3.8-Flash-Next on 26 August 2026 as “an early look at the Qwen4 architecture”. It is not just a bigger Qwen3.8: it uses new building blocks that make it unusually friendly to home hardware.

Official benchmark numbers, as published with the release:

Vendor benchmarks always flatter the vendor, but even with a discount this is a top-tier open model for coding agents, long documents and vision, which is why it is worth the setup effort.

Why a 125B Model Runs on 32 GB of VRAM

Three things make this possible:

• Only ~6B parameters are used per token. The 512 experts sit in memory, but each token only touches 11 of them. llama.cpp can keep the always-used parts (attention, shared expert, embeddings) on the GPU and leave many experts in system RAM, where the CPU handles them with little speed loss.
• The n-gram table does not need fast memory. The 51B-parameter n-gram embedding is looked up, not multiplied: each token reads only 16 rows (a few KB). It can live in system RAM, and on Macs one tester even left most of it on the SSD through llama.cpp’s memory mapping.
• A small KV cache. Because most layers are Gated DeltaNet (linear attention) and only every fourth layer is QSA, the KV cache at 256K context is about 8.4 GB, roughly half that of the dense Qwen3.8-27B.

The catch: the total file still has to fit in VRAM + RAM together. That is why system RAM, not your GPU, decides which quant you can run.

Which GGUF Quant Fits Your PC

These are Unsloth’s GGUF sizes from the Hugging Face repo, with Unsloth’s own quality numbers (top-1 token agreement with the full model; higher is better):

How I size it (my rule of thumb, not an official figure): add the GGUF size, the KV cache for your context (a few GB at 64K with q8_0 cache), and 10-15 GB for Windows, drivers and your browser. With 160 GB total (32 VRAM + 128 RAM), Q3_K_XL leaves a healthy margin and Q4_K_XL is the largest sensible choice. On a machine with 96 GB RAM + one 24 GB GPU, testers ran UD-IQ3_XXS successfully. Below about 75-80 GB of total memory, this model is not a good fit; see If Your PC Is Smaller.

New to quant names like UD-Q3_K_XL? My GGUF quantization guide explains them.

Step 1: Install llama.cpp

You need a llama.cpp build from b10660 or newer, because that is when Qwen3.8-Flash-Next support (the qwen4exp architecture) was merged. Newer is better: multi-GPU and QSA fixes kept landing through September and October 2026.

Windows (NVIDIA)

1. Update your NVIDIA driver. RTX 50 cards need a CUDA 12.8+ capable driver; any current Game Ready or Studio driver is fine.
2. Open the llama.cpp releases page and download the Windows CUDA build (named like llama-bNNNNN-bin-win-cuda-13.x-x64.zip) plus the matching cudart-llama-bin-win-cuda-13.x-x64.zip. For RTX 30 and 40 cards the CUDA 12 build also works.
3. Extract both zips into the same folder, for example C:/llama.
4. Open PowerShell in that folder and check it runs: ./llama --version (older builds: ./llama-server.exe --version).

Linux (NVIDIA)

sudo apt-get update
sudo apt-get install build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j

The binaries end up in llama.cpp/build/bin/. On a Mac, use brew install llama.cpp (Metal is built in), or see my MLX vs GGUF guide for the Apple-native route.

Recent llama.cpp builds use one unified command (llama serve, llama cli). If your build only has llama-server and llama-cli, swap the names in the commands below; the flags are the same.

Step 2: Download the GGUF

You can let llama.cpp download the model on first launch with -hf (Step 3), but for a 90-110 GB download I prefer a resumable download first. Install the Hugging Face CLI and grab only the quant you want:

pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "*UD-Q3_K_XL*" --local-dir C:/models/Qwen3.8-Flash-Next-GGUF
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "*mmproj-F16*" --local-dir C:/models/Qwen3.8-Flash-Next-GGUF

Replace UD-Q3_K_XL with UD-Q4_K_XL if you chose the larger quant. The model comes in several numbered parts (for example ...-00001-of-00003.gguf); keep them together in one folder and always point llama.cpp at the first part. The mmproj file is the vision projector and is only needed for images and video. On Linux use a path such as ~/models/Qwen3.8-Flash-Next-GGUF.

Put the model on an NVMe SSD. Loading 90+ GB from a hard drive or a slow USB disk is the main reason the first start can take several minutes.

Step 3: Launch on 2x 16 GB GPUs + 128 GB RAM

Recommended: let –fit do the split

Modern llama.cpp can work out the best split between your GPUs and system RAM by itself with --fit. One tester with an RTX 4090 and 96 GB RAM found it beat their hand-tuned expert offload. Start here (PowerShell, one command split over lines with backticks):

./llama serve `
  -m C:/models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf `
  --mmproj C:/models/Qwen3.8-Flash-Next-GGUF/mmproj-F16.gguf `
  --fit on `
  -fa on `
  -c 65536 `
  -ctk q8_0 -ctv q8_0 `
  -np 1 `
  --jinja `
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 `
  --port 8080

Check the exact file name in your download folder (the folder layout and number of parts can differ between quants) and adjust the -m path. Or skip the manual download and use -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL instead of -m and --mmproj; llama.cpp then downloads the parts and the vision file into its cache.

What each flag does:

Manual split (if –fit is missing or you want control)

If your build has no --fit, or you want to tune by hand, put all layers on the GPUs, split them evenly, and send some MoE expert layers to the CPU:

./llama serve `
  -m C:/models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf `
  -ngl 99 --tensor-split 1,1 --n-cpu-moe 36 `
  -fa on -c 65536 -ctk q8_0 -ctv q8_0 -np 1 --jinja

--n-cpu-moe 36 keeps the experts of 36 of the 48 layers in RAM. That number is my starting estimate for 32 GB of VRAM, not a measured optimum: if you get a CUDA out-of-memory error, raise it (40, 44); if both GPUs show several GB free in Task Manager, lower it a few at a time for more speed. The model loads once, so each test costs you a reload of a minute or more.

Linux version

The same flags work on Linux. Put the command on one line (or continue lines the usual Bash way), run it from llama.cpp/build/bin/, and use Linux paths such as ~/models/Qwen3.8-Flash-Next-GGUF/.... On dual-socket servers, community users also add --numa distribute.

Step 4: Test It and Pick the Right Settings

When the log says the server is listening, open http://localhost:8080 for llama.cpp’s built-in chat page. The first reply after a fresh load can be slow while the OS pulls the model files into memory; on my PC the first answer after loading takes a few minutes, then it settles down. A safe first test, adapted from a good checklist by aifreeapi.com:

1. Check the model loaded: open http://localhost:8080/v1/models in a browser and confirm the Flash-Next file name is listed.
2. Send one short prompt and watch Task Manager (Performance tab): both GPUs should show high dedicated memory use and RAM should stay below about 90%.
3. Note the speed the chat page shows, then try a longer prompt. If RAM hits 100% and the disk is busy, Windows is swapping: use a smaller quant or a shorter context.
4. If you use tools or agents, test one tool call before connecting a whole agent.

Thinking vs non-thinking mode

Thinking is on by default and the default reasoning effort is xhigh, which can produce very long reasoning. Qwen’s recommended settings:

• Lower the reasoning effort for everyday chat: add --reasoning-effort medium (or low) to the launch command. Note that Qwen warns lower effort can make multi-turn agent tasks slower overall, because the model needs more turns.
• Turn thinking off per request from an API client by sending "chat_template_kwargs": {"enable_thinking": false} in the request body, and switch to the non-thinking sampling above.
• A high presence penalty can cause language mixing, so keep it at 0 in thinking mode.

Any OpenAI-compatible app can now use http://localhost:8080/v1 as its base URL (API key can be any text). That is how I connect it to VS Code agents, as in my Blender MCP with a local LLM guide.

Using Images and Video

Qwen3.8-Flash-Next has a built-in vision encoder, and llama.cpp’s support includes it. Load the mmproj file with --mmproj (or let -hf fetch it), then drag an image into the built-in chat page or send it as an image_url in an OpenAI-style request. The vision projector needs extra memory on top of the model, so drop it (--no-mmproj with -hf) when you only need text and memory is tight. For other local vision options, see my best local vision and video LLMs roundup.

Optional: MTP for Faster Generation

The model ships with a 4B multi-token prediction (MTP) module that drafts several tokens ahead, which the model then checks. Unsloth quotes 1.3-1.7x faster generation with no accuracy loss. It is optional and a bit fiddly, so get the normal setup working first.

1. Download a draft file from the MTP folder of the same repo. The shared Q8_0 file is 2.79 GB and the shared Q4_K_M is 1.91 GB:

   hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "*mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf*" --local-dir C:/models/Qwen3.8-Flash-Next-GGUF

2. Add these flags to your launch command:

   -md C:/models/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3

Things to know: Unsloth’s guide builds MTP from Daniel Han’s qwen4exp/mtp branch, while recent mainline builds also accept MTP drafts; community speed results vary between builds. If the draft model causes a CUDA out-of-memory error on 32 GB, add -ngld 0 to keep the draft in RAM, or use the smaller Q4_K_M draft. One tester found acceptance improved when the draft and main model used similar bit depths.

Ollama and LM Studio

If you prefer a GUI or already use Ollama, both can run the Unsloth GGUF:

ollama run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL

In LM Studio, search for Qwen3.8-Flash-Next and pick an Unsloth GGUF quant. Two honest caveats. First, you get less control over expert offload and the KV cache than with raw llama.cpp, and on this model that matters: one user on Hugging Face reported about 3 tok/s in LM Studio against 30-40 tok/s in a tuned llama.cpp setup on the same machine. Second, the apps bundle their own llama.cpp, so make sure they are fully updated. For a full comparison of the three apps, see Ollama vs LM Studio vs llama.cpp. To set Ollama up from scratch, follow my Ollama install guide.

What Speed to Expect

I am not going to quote a made-up tok/s number for my own PC. Here are published community results on comparable hardware, so you know what is normal:

A 2x 16 GB + 128 GB PC sits in the same class as the first two rows, so expect roughly 20-40 tok/s depending on quant, context, RAM speed and how much ends up on the GPUs (my estimate from these reports). Generation slows as the context fills, which is normal.

If Your PC Is Smaller

• One 16 GB GPU + 96 GB RAM: UD-Q2_K_XL or UD-IQ3_XXS can work with --fit on and a 16-32K context. Expect lower speed than on two GPUs.
• One 16 GB GPU + 64 GB RAM or less: Flash-Next does not fit well. Run the dense Qwen3.8-27B on a 16 GB GPU instead; it scores within a point of Flash-Next on SWE-bench Pro.
• 8-12 GB GPU: pick from my best Ollama models by VRAM list.
• Not sure how much memory you need for other models? See how much VRAM do I need for AI.

Troubleshooting

“unknown model architecture: qwen4exp”

Your llama.cpp (or the copy inside Ollama or LM Studio) is older than build b10660. Download the latest release, or update the app. Also make sure you point at the first GGUF part, not a later one.

CUDA out of memory while loading

In order of what to try: add -np 1 if you forgot it; lower -c (65536 to 32768); keep the q8_0 KV cache; with the manual split, raise --n-cpu-moe by 4; drop --mmproj if you do not need vision; and close other GPU apps (ComfyUI, games, browsers with hardware acceleration). Windows itself uses some VRAM on the GPU your monitor is plugged into, so that card has less free memory than the second one. With a manual split you can give it a smaller share, for example --tensor-split 0.9,1.

Very slow (1-5 tok/s)

• Swapping: if RAM is at 100% and the disk is busy, the model does not fit. Use a smaller quant or shorter context.
• Wrong app settings: GUI apps with default offload can be ten times slower on this model. Try raw llama.cpp with --fit on.
• Memory mapping on Windows: the first replies after loading are slow while files are paged in. If it never speeds up, make sure the model is on an internal NVMe drive and that antivirus is not scanning the model folder.
• Too many CPU threads: on CPUs with performance and efficiency cores, set -t to the number of performance cores.

It thinks for ages before answering

That is the default xhigh reasoning effort. Use --reasoning-effort medium or low, or turn thinking off per request with enable_thinking: false.

MTP draft fails to load

Several users with large setups hit out-of-memory errors on the draft model. Keep the draft in RAM with -ngld 0, use the Q4_K_M draft, reduce --spec-draft-n-max to 2, or run without MTP.

Crashes or odd output at very long context

Some early users saw instability past roughly 80-90K tokens on older builds. Update llama.cpp first. If it still happens, keep one slot (-np 1) and step the context up gradually instead of starting at 262K.

Random Chinese words in English answers

This is usually a high presence penalty or too-low quant. Set presence penalty to 0 (thinking mode) or 1.5 at most (non-thinking), and move up from 1-2 bit quants to Q3 or higher.

Frequently Asked Questions

Can I run Qwen3.8-Flash-Next on a single 16 GB GPU?

Yes, if you have enough system RAM. The GPU only holds part of the model; the rest sits in RAM. With 96 GB or more RAM, a 2-bit or 3-bit GGUF works. With 64 GB or less it does not fit well, so use Qwen3.8-27B instead.

How much RAM do I need for Qwen3.8-Flash-Next?

The GGUF files range from 66.4 GB (UD-IQ1_S) to 111 GB (UD-Q4_K_XL), plus KV cache and room for the OS. Plan on at least 75-80 GB of combined VRAM and RAM; 128 GB of RAM with a 16-32 GB GPU is the comfortable home setup.

Which quant is best for 32 GB VRAM and 128 GB RAM?

UD-Q3_K_XL (89.5 GB) is the comfortable choice. UD-Q4_K_XL (111 GB) gives better quality and still fits with a 32-64K context and few other apps open.

Does llama.cpp support Qwen3.8-Flash-Next?

Yes. Mainline llama.cpp has supported it since build b10660 (PR 27742, merged in late August 2026), including vision through the mmproj file. Use a recent build for the latest multi-GPU and attention fixes.

Can I run Qwen3.8-Flash-Next in Ollama?

Yes, through the Unsloth GGUF: ollama run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL. For the best speed on consumer GPUs, raw llama.cpp with –fit on gives you more control.

Is Qwen3.8-Flash-Next better than Qwen3.8-27B?

On Qwen’s published benchmarks it is slightly ahead on SWE-bench Pro (62.5% vs 61.7%) and clearly ahead on agentic coding (DeepSWE 58.7% vs 42.2%) and multilingual coding. The 27B is much easier to run on a single GPU.

Can I use Qwen3.8-Flash-Next commercially?

It uses the qwen-community-1.0 licence, not Apache 2.0. Read the licence file in the Hugging Face repo before using it in a commercial product.

Final Thoughts

Qwen3.8-Flash-Next is proof that “how much VRAM?” is the wrong question for modern MoE models. With 128 GB of system RAM, two 16 GB cards are enough to run a model that trades blows with frontier systems on coding benchmarks. Start with UD-Q3_K_XL, a recent llama.cpp and --fit on, get a stable 64K context, then experiment with Q4_K_XL, longer context and MTP. If your machine is smaller, the Qwen3.8-27B guide is the better starting point.

Related guides: use it as a coding agent in VS Code with Blender MCP, GGUF quantization explained, Ollama vs LM Studio vs llama.cpp and GLM-4.7-Flash locally for a lighter MoE alternative.

Sources

• Qwen/Qwen3.8-Flash-Next model card (architecture, sampling, licence)
• unsloth/Qwen3.8-Flash-Next-GGUF (file sizes, Ollama command)
• Unsloth docs: Qwen3.8-Flash-Next how to run locally (quants, MTP)
• llama.cpp PR #27742: add Qwen3.8-Flash-Next (qwen4exp)
• DEV Community: llama.cpp support status and A100 benchmarks
• GitHub gist: RTX 4090 + 96 GB llama.cpp config and speeds
• Hugging Face discussion: community MTP and multi-GPU results
• lilting.ch: Flash-Next on M1 Max 64 GB with the n-gram table on SSD
• aifreeapi.com: memory floor and safe first test
• ai-tldr.dev: release date and official benchmarks