How to Run Gemma 4 Locally (2026): Which Size Fits Your GPU, From E2B to 31B

Gemma 4 is Google’s open model family, and since April 2026 it ships under the plain Apache 2.0 licence instead of the old custom Gemma terms. That alone makes it one of the most useful families to run locally. The harder question is which of the five sizes to download, because they are very different: two tiny “E” models for phones and laptops, a 12B added in June 2026, a 26B mixture-of-experts that runs almost as fast as a 4B model, and a dense 31B that is the strongest of the lot. This guide gives the real download sizes, shows which size fits 8, 12, 16 and 24 GB of VRAM, and walks through Ollama, llama.cpp and LM Studio with the settings Google recommends.

What Gemma 4 Is (and What Changed)

Google DeepMind released Gemma 4 on 2 April 2026 in four sizes (E2B, E4B, a 26B mixture-of-experts and a 31B dense model) and added the Gemma 4 12B on 3 June 2026. Quantization-aware-trained (QAT) checkpoints followed in June. The things that matter for local use:

• Apache 2.0 licence for all sizes. Gemma 2 and 3 used custom Gemma terms; Gemma 4 dropped them.
• Multimodal input. Every size reads images with variable aspect ratios. E2B, E4B and the 12B also take audio.
• Long context: 128K tokens on E2B and E4B, 256K on 12B, 26B A4B and 31B.
• Built-in thinking mode, function calling, structured JSON output and a native system role.
• Day-one support in Ollama, llama.cpp, LM Studio, Hugging Face Transformers and MLX.

The Five Gemma 4 Sizes Explained

The “E” in E2B and E4B stands for effective parameters: a large part of each model is embedding tables that can sit in slower memory, so the compute cost is like a 2B or 4B model even though the files are bigger. The “A4B” in 26B A4B means about 4B active parameters per token. You still need memory for all 25.2B weights, but each token only touches 3.8B of them, which is why it is so fast.

Which Gemma 4 Size Fits Your GPU

These are the download sizes on the official Ollama library page (the range covers its default and higher-precision tags) and the sizes of Unsloth’s QAT 4-bit GGUFs, plus Unsloth’s memory guide (RAM + VRAM or unified memory, at about 8K context):

Context length costs memory on top of the weights. Unsloth notes that the full 256K context on the 26B A4B adds about 5 GB of KV cache, so on a 16 GB card start at 16K-32K and raise it only if you need to. My GGUF quantization guide explains how weights and KV cache add up.

My picks by hardware

• CPU only, 8-16 GB RAM: E2B for speed, E4B for quality. Both are fine for summaries, quick questions and image captions.
• 6-8 GB GPU (RTX 3060 Ti, 4060, 5060): E4B with a long context, or the 12B QAT GGUF at 8K-16K context.
• 12 GB GPU (RTX 3060 12 GB, 4070, 5070): 12B. It has the 256K window and audio input, and leaves space for a decent context.
• 16 GB GPU (RTX 4060 Ti 16 GB, 4080, 5070 Ti, 5080): 26B A4B. Because only 3.8B parameters are active per token, it generates far faster than a dense 27B-31B model, and if a few layers spill into RAM it slows down much less than a dense model would.
• 24 GB+ (RTX 3090, 4090, 5090) or two 16 GB cards: 31B. It tops every Gemma 4 benchmark below. On my two RTX 5080s, llama.cpp splits it across both cards.

How Good Is Each Size? Official Benchmarks

Google’s instruction-tuned results, as listed on the Ollama model page (higher is better unless noted):

Two things stand out. First, the 26B A4B is very close to the 31B on knowledge, maths and science, and only clearly behind on long-context retrieval (MRCR) and competitive coding. Second, even the E4B beats last year’s Gemma 3 27B on MMLU Pro, AIME and coding. The 12B was released later and is not in this table; by size you can expect it to land between E4B and 26B A4B, so test it on your own tasks.

Run Gemma 4 With Ollama

Ollama is the easiest route. If you do not have it yet, follow my Ollama install guide for Windows, Mac and Linux, then:

# pick one
ollama run gemma4:e2b
ollama run gemma4:e4b
ollama run gemma4:12b
ollama run gemma4:26b
ollama run gemma4:31b

Ollama handles the chat template for you. To give it an image, type the path in the chat (describe ./photo.jpg) or send it through the API’s images field. To raise the context length (the default is smaller than 256K to save memory), set it for the session or the server:

# inside ollama run
/set parameter num_ctx 32768

# or for every model, before starting the server
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Check where the model ended up with ollama ps. “100% GPU” is what you want; a CPU/GPU split means it did not fit and will be slower. All other commands are in my Ollama commands cheat sheet.

Watch out for cloud tags. gemma4:cloud and gemma4:31b-cloud run on Ollama’s servers, not on your PC. Use them only if you are happy for prompts to leave your machine.

Run Gemma 4 With llama.cpp (QAT GGUF)

For the smallest files and the most control, use Google’s QAT checkpoints converted by Unsloth. Unsloth says the 4-bit QAT format uses about 72% less memory with near-original quality, and that higher precisions did not improve accuracy, so each repo has a single recommended file, UD-Q4_K_XL:

• unsloth/gemma-4-E2B-it-qat-GGUF (2.62 GB)
• unsloth/gemma-4-E4B-it-qat-GGUF (4.22 GB)
• unsloth/gemma-4-12B-it-qat-GGUF (6.72 GB)
• unsloth/gemma-4-26B-A4B-it-qat-GGUF (14.2 GB)
• unsloth/gemma-4-31B-it-qat-GGUF (17.3 GB)

Quick start, letting llama.cpp download the file itself:

llama-cli -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64

For images you also need the vision projector (mmproj-BF16.gguf) from the same repo. Download both, then start an OpenAI-compatible server:

hf download unsloth/gemma-4-26B-A4B-it-qat-GGUF --local-dir gemma4-26b --include "*mmproj-BF16*" --include "*UD-Q4_K_XL*"

llama-server --model gemma4-26b/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --mmproj gemma4-26b/mmproj-BF16.gguf -ngl 99 -c 32768 --temp 1.0 --top-p 0.95 --top-k 64 --port 8001 --chat-template-kwargs '{"enable_thinking":true}'

Set enable_thinking to false to switch thinking off (on Windows PowerShell, escape the quotes). Recent llama.cpp builds also offer the unified llama serve and llama cli commands; the options are the same. For the 31B on two 16 GB GPUs, llama.cpp spreads layers across both cards automatically; add --tensor-split 1,1 to make the split even.

If you are not sure whether llama.cpp, Ollama or LM Studio suits you better, my Ollama vs LM Studio vs llama.cpp comparison covers speed, VRAM use and features.

Run Gemma 4 in LM Studio and on a Mac

• LM Studio: search “gemma-4” in the Discover tab, pick the size, and choose a 4-bit GGUF (the QAT file if listed). Load it, set the context length in the right-hand panel, and turn on GPU offload to the maximum your VRAM allows.
• Mac (Apple Silicon): Ollama has MLX tags for every size (gemma4:e2b-mlx 7.5 GB, e4b-mlx 9.5 GB, 12b-mlx 7.7 GB, 26b-mlx 18 GB, 31b-mlx 19 GB). LM Studio offers MLX versions too. A 16 GB Mac runs the 12B well; the 26B A4B needs a 24-32 GB Mac.

Best Settings: Sampling, Thinking, Images and Context

Thinking makes answers slower but much better on maths, logic and multi-step coding. For quick chat, captions or translation, leave it off.

Gemma 4 vs Qwen3.8 and Other Local Models

Gemma 4 is not the only strong open family this year. In my best Ollama models ranking, Qwen3.8 27B is the best all-rounder on 16-24 GB, Gemma 4 31B is the best for reasoning and maths, and Gemma 4 12B is the pick for 12 GB cards that need vision. Choose Gemma 4 when you want:

• A clean Apache 2.0 licence for a product.
• Audio input in a small model (E2B, E4B, 12B).
• A fast MoE for 16 GB (26B A4B) with strong maths and science scores.
• Good multilingual results (MMMLU 88.4% on the 31B).

For vision-heavy work, compare it with the models in my local vision and video LLM roundup.

Common Problems and Fixes

Frequently Asked Questions

Can I run Gemma 4 locally?

Yes. All five sizes are open weights under Apache 2.0 and run in Ollama (ollama run gemma4:12b), llama.cpp, LM Studio and MLX. The E2B even runs on a CPU.

How much VRAM does Gemma 4 need?

At 4-bit with about 8K context: E2B about 3 GB, E4B 5 GB, 12B 7 GB, 26B A4B 15 GB and 31B 18 GB of combined RAM and VRAM, according to Unsloth’s guide. Longer context needs more.

Which Gemma 4 model is best for a 16 GB GPU?

The 26B A4B mixture-of-experts. Its QAT GGUF is 14.2 GB and only 3.8B parameters are active per token, so it is fast. Use the 12B if you need audio input or a very long context.

Is Gemma 4 free for commercial use?

Yes. Gemma 4 is released under the Apache 2.0 licence, unlike Gemma 2 and 3, which used custom Gemma terms.

Does Gemma 4 support images and audio?

All sizes accept images. E2B, E4B and the 12B also accept audio. Whether your app supports audio depends on the runtime, so check its release notes.

What are the best settings for Gemma 4?

Temperature 1.0, top_p 0.95 and top_k 64. Enable thinking with the think token at the start of the system prompt for maths, logic and coding.

What is the difference between Gemma 4 26B A4B and 31B?

The 26B A4B is a mixture-of-experts with 25.2B total and 3.8B active parameters, so it is much faster. The 31B is dense and scores higher, especially on long-context retrieval and competitive coding.

Final Thoughts

Gemma 4 has a size for almost every machine: E2B and E4B for CPUs and laptops, 12B for 8-12 GB cards, 26B A4B as the sweet spot for 16 GB, and 31B for 24 GB and dual-GPU rigs. Start with Ollama, switch to the QAT GGUFs in llama.cpp when you want to squeeze more into your VRAM, and keep temperature at 1.0. If you outgrow it, see how much hardware the big models need in my Kimi K3 VRAM guide, or try a small but strong coder like MiMo-V2.6 9B.

Sources

• Ollama library: gemma4 (tags, sizes, benchmarks, settings)
• Unsloth docs: Gemma 4 QAT (GGUF sizes, memory, llama.cpp commands)
• Gemma 4 release under Apache 2.0 (April 2026)
• Wikipedia: Gemma (release timeline incl. Gemma 4 12B)
• OrcaRouter: Gemma 4 sizes from the model cards
• Google DeepMind: Gemma