vLLM vs llama.cpp vs Ollama (2026): Benchmarks, Hardware and Which to Use

vLLM, llama.cpp and Ollama all run open models on your own hardware, but they are built for very different jobs. Pick the wrong one and you either fight a server tool for a simple chat, or watch a home tool collapse when ten people use it at once.

This guide compares them in plain terms: what each one is for, real benchmark numbers from October 2026, hardware and OS support, model formats, and the exact commands to get each one serving an OpenAI-compatible API.

Quick Comparison Table

What Each Tool Actually Is

Ollama

Ollama is a friendly app and background service. You type ollama run with a model name, and it downloads, loads and chats in one step. It uses llama.cpp and MLX under the hood and adds its own model library.

Recent releases keep adding features. v0.40.11 (24 October) added speech generation on macOS with an OpenAI-style /v1/audio/speech endpoint, and v0.40.12 (27 October) added Liquid AI LFM3 and updated MLX and llama.cpp.

llama.cpp

llama.cpp is the C/C++ inference engine that made local LLMs popular. It gives you llama-cli for chat and llama-server for an API with a built-in web UI. It releases several builds a day, numbered like b12017.

It has the widest hardware support of the three and the finest control over where each part of a model lives, which is why it is the go-to for running big models on modest GPUs.

vLLM

vLLM is a high-throughput serving engine from the Berkeley team behind PagedAttention. It uses continuous batching, chunked prefill and prefix caching to serve many requests at once with little wasted VRAM.

It is what companies and cloud providers use to serve open models. The latest release, v0.33.0 (2 November 2026), ships CUDA 13.0-only wheels and adds new models such as Step-5 Preview.

For a wider look at desktop apps, see my Ollama vs LM Studio vs llama.cpp comparison.

Benchmarks: One User vs 32 Users

These numbers are from a public test by ComputingForGeeks in October 2026, using Qwen3.8-27B at 4-bit on Ubuntu 24.04 with CUDA 13.2. I did not run these myself, but they match what the tools are designed for.

What this shows:

• For one user, all three are close. With its default MTP draft (speculative decoding) on, Ollama was actually the fastest single-user option: 129.7 tok/s on the RTX 5090 and 76.2 on the 4090.
• With 32 users, vLLM wins clearly on bigger GPUs: 936 tok/s on the 5090, about 2.5x llama.cpp.
• Ollama served one request at a time for this model, so 32 users meant a long queue (54-200 seconds before the first word).
• vLLM with MTP reached 135.3 tok/s single-user on the 5090, but it would not start with MTP on the 24 GB 4090.

Hardware and OS Support

The big practical point: vLLM is built for Linux. On Windows you run it inside WSL2 or a Docker container. Ollama and llama.cpp both have native Windows builds.

Model Formats and Quantization

• Ollama and llama.cpp use GGUF files with quants like Q4_K_M, Q5_K_M, Q8_0 and IQ3/IQ4. One file holds the whole model, and they come in many sizes so you can fit almost any GPU.
• vLLM loads Hugging Face safetensors checkpoints. It supports FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, compressed-tensors and more. It can load GGUF, but that is not its strong side.

For vLLM, look for repositories labelled FP8, AWQ, GPTQ or W4A16 (for example from RedHatAI or the model maker). For Ollama and llama.cpp, Unsloth and bartowski GGUF repositories are the usual choice. My best Ollama models list has good starting picks.

VRAM, RAM Offload and Big MoE Models

This is where the tools differ most for home users.

• vLLM wants the whole model plus its KV cache in VRAM. By default it reserves 90% of GPU memory (the --gpu-memory-utilization setting) and fills the spare space with cache for many users.
• llama.cpp can split a model between GPU and RAM layer by layer with -ngl, and it can keep MoE experts in RAM with --cpu-moe or --n-cpu-moe while attention runs on the GPU. That is how people run 100B+ MoE models on a 16-32 GB card.
• Ollama offloads automatically. It is easy, but you get less control than raw llama.cpp.

If you want to run something like Qwen3.8-Flash-Next on a desktop, llama.cpp is the tool. See my Qwen3.8-Flash-Next local guide, plus how much VRAM you need and how much RAM you need for local AI.

How to Start Each One (Commands)

Ollama

1. Install it from ollama.com (full steps in my Ollama install guide).
2. Run a model: ollama run <model>, using any tag from the Ollama library page (for example a 4-bit 27B model on a 24 GB card).
3. The API is already running at http://localhost:11434. The OpenAI-style endpoint is http://localhost:11434/v1.
4. To allow more than one request at once (for models that support it), set the environment variable OLLAMA_NUM_PARALLEL, for example to 4, and restart Ollama.

llama.cpp

1. Download a prebuilt release from the llama.cpp GitHub releases page. On Windows with NVIDIA, pick the Windows x64 (CUDA 13) or (CUDA 12) zip. Use Vulkan for AMD or Intel if ROCm or SYCL give you trouble.
2. Download a GGUF file from Hugging Face.
3. Start the server with all layers on the GPU and 4 parallel slots:
   llama-server -m model.gguf -ngl 99 -c 32768 -np 4 --port 8080
4. Open http://localhost:8080 for the built-in chat UI, or use http://localhost:8080/v1 as an OpenAI API.
5. For a big MoE model, add --n-cpu-moe 20 (raise or lower the number until it fits your VRAM).

The context size set with -c is shared between the -np slots, so 32768 with 4 slots gives each user about 8K tokens.

vLLM (Linux or WSL2)

1. Create a Python environment, then install with uv as the README shows:
   uv pip install vllm
2. Optionally download precompiled kernels for faster startup: vllm download-kernels
3. Serve a model from Hugging Face:
   vllm serve <org>/<model-FP8> --max-model-len 32768, using the exact Hugging Face repository name of the checkpoint you want
4. The OpenAI-compatible API is at http://localhost:8000/v1.
5. For two GPUs, add --tensor-parallel-size 2.

OpenAI-Compatible API and Front Ends

All three speak the OpenAI API, so the same apps work with each. Point the app’s “base URL” at the right address:

For a ChatGPT-style interface on top, use Open WebUI. My Open WebUI setup guide shows the Ollama route, and the same “OpenAI connection” setting works for llama.cpp and vLLM.

Which One Should You Use?

If you are renting hardware for vLLM, my cheapest cloud GPU for AI guide compares prices.

Troubleshooting

vLLM: CUDA out of memory at startup

vLLM reserves most of the GPU for cache. Lower --max-model-len (for example to 16384), lower --gpu-memory-utilization to 0.85, or pick a smaller quant. Close other GPU apps first. More fixes in my CUDA out of memory guide.

vLLM will not install on Windows

vLLM targets Linux. Install WSL2 with Ubuntu, install the NVIDIA driver on Windows (not inside WSL), then install vLLM inside WSL. Or run the official Docker image.

vLLM: errors after upgrading to v0.33

The wheels need CUDA 13.0. Update the NVIDIA driver to a version that supports CUDA 13, and make sure you are not on Python 3.10.

Ollama gets very slow with several users

Some models only serve one request at a time in Ollama. Set OLLAMA_NUM_PARALLEL, and if the model still queues, move that workload to llama.cpp or vLLM.

llama.cpp is slow or uses the CPU

You probably downloaded the CPU build or forgot -ngl. Use a CUDA, ROCm or Vulkan build and add -ngl 99. The startup log shows how many layers went to the GPU.

llama.cpp: “failed to allocate” or crashes when loading

Lower -ngl, reduce -c, or for MoE models raise --n-cpu-moe. Check your free VRAM with my GPU VRAM usage guide.

Answers look broken or repeat

Usually a chat template issue. Use the GGUF or checkpoint from the official or a trusted repository, update to the latest build, and avoid very old quants made before a model’s template was fixed.

My Take

For a home PC, start with Ollama. It is the quickest way from zero to a working chat, and with its speculative decoding it is very fast for one person.

Move to llama.cpp when you want to squeeze a big model onto your hardware or share one GPU with a few people. It is the most flexible engine there is.

Choose vLLM when you are serving real traffic: a team, an agent framework firing many calls, or a product. It needs Linux and enough VRAM, but nothing else matches its throughput.

FAQ

Is vLLM faster than Ollama?

For many users at once, yes, by a lot: 936 vs 74 tok/s total at 32 users on an RTX 5090 in one October 2026 test. For a single user they are close, and Ollama with its default speculative decoding can be faster.

Is Ollama just llama.cpp?

Ollama uses llama.cpp (and MLX on Mac) as engines, but adds its own model library, automatic offloading, a background service and a simple CLI and API.

Does vLLM work on Windows?

Not natively. Run it in WSL2 or Docker on Windows. Ollama and llama.cpp have native Windows builds.

Can vLLM run GGUF models?

It can load GGUF, but it works best with safetensors checkpoints in FP8, AWQ, GPTQ or other formats it lists. For GGUF, llama.cpp or Ollama are the better fit.

Which is best for running models bigger than my VRAM?

llama.cpp. It can keep MoE experts in system RAM with --n-cpu-moe while the GPU handles the rest. vLLM expects the model to fit in VRAM.

Which one should I use with an AMD GPU?

llama.cpp is the safest choice, with ROCm and Vulkan builds for Windows and Linux. Ollama also supports many AMD cards, and vLLM supports ROCm on Linux.

Do all three have an OpenAI-compatible API?

Yes. Ollama serves it at port 11434, llama-server at 8080 and vLLM at 8000, all under /v1.

Sources

• ComputingForGeeks: Ollama vs vLLM vs llama.cpp benchmarks (October 2026)
• vLLM releases on GitHub (v0.33.0, 2 Nov 2026)
• vLLM README (install, hardware, quantization)
• Ollama releases on GitHub (v0.40.12)
• llama.cpp releases on GitHub (b12017 builds)