Skip to content
, ,

Edge0-145B: How to Run a 145B Open Model on One GPU (Setup, VRAM, Speeds, Licence)

12–18 minutes
Woman at a gaming PC holding a tablet with an Edge0 card, hook 145B model on one GPU

A 145-billion-parameter model that runs on one gaming GPU sounded like a stretch a year ago. On 12 November 2026 the Edge0 team open-sourced Edge0-145B and a smaller Edge0-120B, both built to run on a single NVIDIA card with help from system RAM and your SSD.

The headline claims are big: 80+ tokens per second, about 29 GB peak memory, and agent-coding scores that beat Qwen3.8-Flash-Next, the model it is built on. All of those numbers come from Edge0 itself, so this guide separates what is measured from what is marketing.

Below you will find the real hardware needs, which file to download, how to set it up on Windows or Linux, the speed tables by GPU, the licence catch, and how it compares with the other big local models.

TL;DR

  • What: Edge0-145B (145B total, 10.12B active per token) and Edge0-120B (8.64B active), sparse MoE models fine-tuned and rebuilt from Qwen3.8-Flash-Next. Released 12 Nov 2026.
  • Runs on: one NVIDIA GPU (RTX 20-series or newer) with CUDA 13 on Windows or Linux, using the Edge0 runtime edge0-infer. Edge0 tested a 16 GiB VRAM + 20 GiB RAM budget.
  • Download size: 40.81 GiB (Q3_K) or 47.79 GiB (NVFP4) for 145B; 27.98 / 32.96 GiB for 120B.
  • Speed (vendor-reported): 69.9 tok/s chat on an RTX 4090, 99.7 tok/s on an RTX 5090 D, 36.8 tok/s at 16 GiB VRAM.
  • Licence: free for personal, research and internal use, but running a commercial model service needs a written licence from Edge0.
  • Catch: the “29 GB” figure is not the full footprint, Ollama and LM Studio support is not confirmed, and no independent benchmarks exist yet.

What Edge0-145B Is and Why It Matters

Edge0 is the on-device AI team behind the Edge0-35B phone model and the Audio8 TTS family. Its blog post of 12 November calls Edge0-145B “the world’s first open-source on-device AI model with 145 billion parameters”.

The trick is in how the weights are stored. Only about 10B parameters work on each token, so the GPU only needs the “hot” experts. The rest sit in system RAM, and a huge 51.2B n-gram embedding table is memory-mapped from your SSD and read on demand.

Edge0 says the models were designed for local use from the start, not squeezed down after cloud training. The Hugging Face card lists Qwen/Qwen3.8-Flash-Next as the base model, so think of it as a heavy rebuild of Qwen’s MoE for single-GPU machines.

Why local AI fans care

  • One GPU, not four: Qwen3.8-Flash-Next at 4-bit needs about 70 GB by Edge0’s own estimate. Edge0-145B targets a 16-24 GB card plus RAM.
  • Agent coding focus: the claimed Terminal-Bench 2.1 score of 86.52 is higher than Claude Opus 4.6 Max in Edge0’s table.
  • OpenAI and Anthropic APIs locally: the server speaks both formats, so tools like Claude Code-style agents and Cline can point at it.

Specs: 145B vs 120B

Both models share the same 48-layer design. The 120B version has a smaller expert pool, which makes it faster and lighter on RAM.

Item Edge0-145B Edge0-120B
Total / active parameters 145B / 10.12B 120B / 8.64B
Experts per layer 512 (top-10 routed + 1 shared) 512 (top-10 + 1 shared)
Expert pool 84.6B 60.4B
Layers 48 (full attention every 4th layer + sparse indexer) 48
Context window 256K (int8 KV, 32K resident) 256K
Speculative decoding 1.26B MTP draft head 1.26B MTP draft head
N-gram embedding (PLE) 51.2B, fp8, memory-mapped 51.2B, fp8, memory-mapped
Q3_K weights 40.81 GiB 27.98 GiB
NVFP4 weights 47.79 GiB 32.96 GiB
Vision Yes, needs separate strata-vision helper Same

Note: the runtime defaults to a 131,072-token context on Windows and Linux. The 256K figure is the model limit, not the shipping default.

Hardware You Need (and the 29 GB Question)

The Edge0 Chat v0.2.0 release notes give the clearest minimums for the 145B and 120B models on Windows x64:

  • GPU: NVIDIA with compute capability 7.5 or higher, so RTX 20, 30, 40 or 50 series.
  • VRAM: at least 16 GiB.
  • System RAM: at least 20 GiB free for the model.
  • Software: CUDA Toolkit 13. Linux follows the same rule.
  • Disk: a fast NVMe SSD with room for 41-48 GB of weights.

AMD and Intel GPUs are not supported for these two models. Windows on Arm works only with the RTX Spark N1X chip (sm_121).

What about the “29 GB peak memory” claim?

The launch blog says 29 GB peak. The model card tells a fuller story: with residency on, the 145B Q3_K build used 23.14 GiB VRAM plus 31.28 GiB RAM on an RTX 5090 D. That is about 54 GiB in total, which a Japanese blogger also pointed out after launch.

So 29 GB is roughly the active working set, not everything the process touches. Plan for 16-24 GB of VRAM and 32-64 GB of RAM if you want the speeds in the tables.

Your machine Can it run Edge0-145B? Better pick
RTX 4090 / 5090, 64 GB RAM Yes, full speed tier 145B Q3_K
RTX 4080 / 5080 16 GB, 32-64 GB RAM Yes, in the 16 GiB tier (about 37-45 tok/s per Edge0’s capped test) 120B Q3_K for more speed
2x RTX 5080 16 GB, 128 GB RAM Yes; second GPU can act as an extra expert cache 145B Q3_K
RTX 3060 12 GB Below the official 16 GiB minimum A smaller Qwen model
Mac (Apple Silicon) Q3_K only via Metal, in the Edge0 Chat Mac build; model card says Mac results “do not establish” release support Treat as experimental
AMD Radeon / Intel Arc No (CUDA only) Qwen3.8 GGUF in llama.cpp

Tip: the 16 GiB tier is an estimate for real 16 GB cards. Edge0 tested it by capping an RTX 5090 D, not on an actual RTX 5080, so your numbers may differ. Our RTX 5090 vs RTX 5080 for AI guide explains the bandwidth gap.

Which File to Download: Q3_K or NVFP4

Edge0 publishes four model folders on Hugging Face under the Edge0 organisation. Pick by GPU generation:

Folder Size Best for
Edge0/Edge0-145B-q3k 40.81 GiB Most people: RTX 20-50, and the only build for Mac. Fastest decode in Edge0’s tests.
Edge0/Edge0-145B-NVFP4 47.79 GiB RTX 50 / Blackwell owners who want FP4 math. CUDA only. Needs more RAM (38.27 GiB).
Edge0/Edge0-120B-q3k 27.98 GiB 16 GB cards and 32 GB RAM machines.
Edge0/Edge0-120B-NVFP4 32.96 GiB RTX 50 cards, 120B tier.

Interestingly, Q3_K was faster than NVFP4 on the RTX 5090 D (99.7 vs 79.6 tok/s chat), so most people should start with Q3_K.

Warning: keep the downloaded folder exactly as it is. edge0-infer reads the model type from model.json inside the folder, and renamed or partial folders fail to load.

If you have never pulled a large repo, our guide to downloading models from Hugging Face covers the hf CLI, resuming and cache paths.

How to Run Edge0-145B, Step by Step

There are two routes. The Windows app is easiest; the command line gives you the API server and all memory flags.

Route 1: Edge0 Chat app on Windows

  1. Install CUDA Toolkit 13. Get it from NVIDIA’s CUDA downloads page and update your GeForce driver first. Without it, the app can fail to start with error 0xC0000135 (missing DLL).
  2. Download the app. From the GitHub releases page of Edge0-AI/Edge0, grab edge0-chat-0.2.0-windows-x64.zip (about 264 MB). It includes a prebuilt edge0-infer.exe.
  3. Check the file. The release lists SHA256 c7146b76d51c7986723b81a3643893e1ccbb09ce4f365a5090dc99e476bb31b9. In PowerShell run Get-FileHash .\edge0-chat-0.2.0-windows-x64.zip and compare.
  4. Download the weights. For example: hf download Edge0/Edge0-145B-q3k --local-dir D:\models\Edge0-145B-q3k. Weights are not inside the app zip.
  5. Open the folder in Edge0 Chat. Unzip, start the app, and choose the model directory. First load is slow while experts are read from the SSD.

Route 2: command line and local API (Windows or Linux)

  1. Clone the repo. git clone https://github.com/Edge0-AI/Edge0.git edge0
  2. Build the two engines. Follow the “Build (Linux, NVIDIA)” or “Build (Windows, NVIDIA)” section in edge0-strata/EDGE0.md, then the getting-started guide in edge0-infer/docs. You need both edge0-strata and edge0-infer.
  3. Check the model folder. edge0-infer inspect --model /path/to/Edge0-145B-q3k --json
  4. Start the server. build/cpu/bin/edge0-infer serve --model /path/to/Edge0-145B-q3k (on Windows: build\windows\bin\edge0-infer.exe). Wait for the “Listening” line on http://127.0.0.1:8000.
  5. Chat or connect a tool. Run edge0-infer chat --model /path/to/Edge0-145B-q3k, or point any OpenAI-compatible client at http://127.0.0.1:8000/v1 with model name edge0_145b. Anthropic-style POST /v1/messages also works.

A quick test call from a terminal looks like this:

curl http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"edge0_145b","messages":[{"role":"user","content":"Write a Python function that reverses a string."}],"reasoning_effort":"none"}'

Note: only serve, chat and inspect work with these models. The run and bench commands are for Edge0’s older native engine and refuse the 145B and 120B folders.

Windows users can also build the Linux way inside WSL2; our WSL2 local AI setup guide covers CUDA inside WSL.

Memory Flags That Matter on 16 GB Cards

The model card lists the flags that decide where experts live. These are the useful ones:

Flag Default What it does
--vram-reserve-mib N 700 Keeps VRAM free for Windows and other apps. Edge0’s 16 GiB test used 8192 on a 24 GB card. Not shown in --help.
--expert-cache N|auto 0 Sizes the GPU expert cache; auto uses free VRAM.
--expert-cache-device1..3 N|auto off Uses a 2nd, 3rd or 4th GPU as extra expert cache.
--mmap-experts off Reads uncached experts from SSD on demand instead of RAM.
--resident-budget-gib 20 off Caps the RAM expert cache. Requires --expert-profile <file>.
--kv-resident N 0 Streams KV cache from pinned host RAM.
--mtp-q4 all|proj|head off Shrinks the draft head, saving about 0.3-0.5 GiB VRAM.
--spec N 4 Speculative tokens per step from the MTP head.
--memory-limit-mib N 0 (auto) Host RAM limit for the process.

For a single 16 GB card with 32 GB RAM, a sensible starting line is:

edge0-infer serve --model /path/to/Edge0-120B-q3k --expert-cache auto --vram-reserve-mib 1500 --mtp-q4 all

That is our suggested starting point built from the documented flags, not an Edge0-published recipe. Raise --vram-reserve-mib if your desktop stutters, and switch to the 145B folder once it runs cleanly.

Speed by GPU

All numbers below are vendor-reported from the Edge0 model card (tokens per second). “Agent decode” is faster because the MTP draft head guesses predictable code tokens well.

Machine Build 16K prefill Chat decode Agent decode
RTX 5090 D 24 GB 145B Q3_K 3,554.9 99.7 121.2
RTX 5090 D 24 GB 145B NVFP4 3,165.7 79.6 98.3
RTX 4090 24 GB 145B Q3_K 3,179.4 69.9 91.7
RTX Spark N1X 60 GiB 145B NVFP4 922.4 40.1 46.0
Apple M2 Ultra (Metal) 145B Q3_K 800.3 60.9 57.0
16 GiB VRAM + 20 GiB RAM cap 145B Q3_K 335 36.8 44.9
16 GiB VRAM + 20 GiB RAM cap 120B Q3_K 778 62.4 73.3
RTX 4090 24 GB 120B Q3_K 3,551.1 95.0 104.9

The big takeaway: on a 16 GB card, Edge0-120B is almost twice as fast as 145B (62.4 vs 36.8 tok/s chat) and prefill is over twice as quick. For long agent prompts, prefill speed matters more than decode.

Benchmarks (Vendor-Reported)

Edge0 ran these scores itself. Rows for Flint, Muse Glimmer and Gemma 4 mix in third-party numbers, and a full technical report with the test setup is still “to be released”.

Model DeepSWE 1.1 Terminal-Bench 2.1 HLE GPQA-Diamond IFBench
Edge0-145B 64.6 86.52 35.4 90.4 80.6
Edge0-120B 53.6 81.3 31.0 87.4 79.3
Qwen3.8-Flash-Next 58.7 83.3 35.9 91.7 81.3
DeepSeek-V4-Flash-0731 54.4 82.7 33.8 90.8 79.2
Qwen3.8-27B 42.2 73.0 30.8 89.2 79.5
Claude Opus 4.6 (Max) n/a 78.2 40.0 91.3 62.5

Read it carefully. Edge0-145B wins the two agent coding tests, while its own base model, Qwen3.8-Flash-Next, still edges it on HLE, GPQA and instruction following. Claude Opus 4.6 is also an older Claude model; current Claude models are not in the table.

Edge0-145B vs Other Big Local Models

Model Active params Memory to run (approx.) Runtimes Licence
Edge0-145B 10.12B 16-24 GB VRAM + 20-38 GB RAM (measured by Edge0) edge0-infer, Edge0 Chat Edge0 Community 1.0 (commercial service needs licence)
Edge0-120B 8.64B 16-23 GB VRAM + 18-23 GB RAM edge0-infer, Edge0 Chat Same
Qwen3.8-Flash-Next (4-bit) 6B ~70 GB total (Edge0 estimate) llama.cpp, Ollama, LM Studio, vLLM Qwen Community 1.0
DeepSeek-V4-Flash-0731 (4-bit) n/a ~87 GB total (Edge0 estimate) llama.cpp, vLLM DeepSeek licence
Qwen3.8-27B (4-bit) 27B dense ~18 GB (Edge0 estimate) All major runtimes Qwen Community 1.0

Who should use which

  • One 16-24 GB NVIDIA card, coding agents: try Edge0-145B or 120B. It is the only one in this list built for that setup.
  • You want Ollama or LM Studio convenience: stay with Qwen3.8 models for now. Our Qwen3.8-Flash-Next local guide covers the 2x 16 GB + 128 GB RAM route.
  • AMD, Intel or older Mac: Edge0 is not for you yet. Use llama.cpp; see our llama.cpp 1.0 guide.
  • You sell an AI product: check the licence below before building on Edge0.

Licence: What You Can and Can’t Do

This is not Apache 2.0 for the weights. Three licences apply:

  • Model weights (Edge0 parts): Edge0 Community Model License 1.0.
  • Qwen-derived parts: Qwen Community License 1.0.
  • Code: the framework is Apache-2.0 and the Strata engine is MIT.
Allowed without asking Needs a separate written Edge0 licence
Personal use and research Selling the model or bundling it in a paid product
Modifying and fine-tuning Running a commercial model service (paid API)
Non-commercial sharing A standalone commercial coding or office assistant built on it
Internal use at your company
Commercial use of the outputs you generate

Warning: the Hugging Face page metadata shows “5B params” and architecture “qwen4exp”, which clashes with the card text. Trust the card body and the GitHub docs, and read the full licence file before any business use.

Known Problems and Fixes

  • Empty replies: with thinking on, the model can spend the whole token budget reasoning and return nothing. Add "reasoning_effort": "none" to the request, or raise max tokens.
  • “this server was started without the vision encoder”: image input needs the separate strata-vision helper, which is not in the main repo yet.
  • App won’t start, exit code 0xC0000135: the CUDA 13 runtime DLL is missing. Install CUDA Toolkit 13 and reboot.
  • Error with --resident-budget-gib: this flag needs a static --expert-profile file.
  • Slow first answer: experts load from SSD on the first request. Later requests are faster once the OS page cache is warm. An NVMe drive helps a lot.
  • Converter fails on Windows with WinError 1314: a community user hit this with the older asset converter, which uses symlinks. Turn on Windows Developer Mode or run as admin.
  • Requests seem stuck: the server queues concurrent requests instead of rejecting them, so a second client waits for the first to finish.
  • Different answers at temperature 0: reported in coverage of the runtime guide; do not expect bit-exact repeats.
  • Out-of-memory crashes: raise --vram-reserve-mib, use --mtp-q4 all, or drop to Edge0-120B. Our CUDA out of memory guide has general checks.

Ollama and LM Studio: the Hugging Face page shows generic “Use this model” snippets for llama.cpp, Ollama, vLLM and LM Studio with a :BF16 tag that does not exist in these repos. Edge0’s own docs only document edge0-infer, so treat those snippets as unconfirmed.

My Take

This is the most interesting local model release of the month for people with one good NVIDIA card. The architecture is clever: a tiny active set, experts spread across VRAM, RAM and SSD, and a draft head for speed.

The detailed per-GPU tables and the honest “test budget, not minimum” notes are better than most launches.

But I would not call it proven. Every speed and score is Edge0’s own, the “29 GB” headline hides a ~54 GiB real footprint, and macOS support is muddled between the blog and the docs. The agent wins over Qwen3.8-Flash-Next are also small on some tests.

If you have an RTX 4090, 5090 or a 16 GB card with 64 GB RAM, it is worth an evening to test on your own coding tasks. Start with Edge0-120B Q3_K on 16 GB cards. If you need Ollama-style ease or a commercial product, wait for the technical report and wider runtime support.

FAQ

Is Edge0-145B free?

Yes, the weights are free to download for personal, research and internal use. A commercial model service or paid assistant built on it needs a written licence from Edge0.

Can Edge0-145B run on a 16 GB GPU?

Edge0 lists 16 GiB VRAM and 20 GiB RAM as the minimum and reports 36.8 tok/s in a capped 16 GiB test. That test used a limited RTX 5090 D, so a real 16 GB card may be slower.

Does it work in Ollama or LM Studio?

Not officially yet. Edge0’s documentation only covers its own edge0-infer runtime and the Edge0 Chat app.

Does it run on a Mac?

The Edge0 Chat v0.2.0 Mac build supports the Q3_K folders through Metal on macOS 26 or later. The model card says its M2 Ultra results do not confirm release support, so treat it as experimental.

Should I pick 145B or 120B?

Pick 145B on 24 GB cards for the best scores. On 16 GB cards, 120B is nearly twice as fast and needs less RAM.

Is it better than Claude or GPT for coding?

Edge0’s table puts it ahead of Claude Opus 4.6 Max on Terminal-Bench 2.1, but newer cloud models are not compared and no independent tests exist yet.

Does it support AMD GPUs?

No. The 145B and 120B models need an NVIDIA GPU with CUDA 13. Only the older Edge0-35B runs on Vulkan or CPU.

What is the context length?

The model supports 256K tokens, but the runtime defaults to 131,072 tokens on Windows and Linux and 32,768 on macOS.

Sources

  • Edge0 blog, “Introducing Edge0-145B: Frontier AI Capabilities on Your Own Computer” (12 Nov 2026)
  • Hugging Face model cards: Edge0/Edge0-145B-q3k and Edge0/Edge0-145B-NVFP4 (checked 16 Nov 2026)
  • GitHub, Edge0-AI/Edge0: docs/models/edge0-145b.md and Release 0.2.0 (Edge0 Chat, 11 Nov 2026)
  • WindowsForum, “Edge0-145B on Windows: Local AI Hardware Requirements, RTX 5090 Results and Deployment Limits” (Nov 2026)
  • Jan Sverre, “Edge0-145B on RTX 4090: speed and hardware requirements” (Nov 2026)
  • DEV Community (qianchuan), “Everyone Is Benchmarking Edge0-145B on RTX 4090s” (Nov 2026)
  • FreeAnswers, “Edge0-145B: Local Setup, Hardware, Benchmarks and License” (Nov 2026)
Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Woman at a gaming PC holding a tablet with an Edge0 card, hook 145B model on one GPU