llama-server is the server that ships with llama.cpp. One command turns any GGUF model into a local chat app in your browser plus an OpenAI-compatible API that Open WebUI, coding agents, LangChain and your own scripts can use.
It is lighter than Ollama, gives you every llama.cpp option directly, and usually gets new model support first.
This guide covers installing it on Windows, Mac and Linux, the commands you actually need, the flags that matter for speed and memory, calling the API, running several users or models, and fixing the common problems. Every flag and default here is checked against the current llama.cpp server documentation.
- Install:
winget install llama.cpp(Windows) orbrew install llama.cpp(Mac/Linux), or a prebuilt zip from GitHub Releases. - Run:
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080downloads a model and starts the server. - Open:
http://127.0.0.1:8080for the chat app. The API is at/v1/chat/completions. - Default port is now 9931, not 8080. Always pass
--portso other tools know where to connect. - Memory is automatic:
--fit(on by default) picks GPU layers and context to fit your VRAM. Set-cand-nglyourself only when you want control.
New to running models locally? My Ollama guide is the easiest starting point, and my llama.cpp 1.0 guide explains what changed in the latest release.
llama-server vs Ollama vs LM Studio
| llama-server | Ollama | LM Studio | |
|---|---|---|---|
| Engine | llama.cpp itself | Its own engine plus llama.cpp | llama.cpp and MLX |
| Interface | Command line + built-in web chat | App + CLI | Desktop app |
| Model files | Any GGUF file or Hugging Face repo | Ollama library (or imported GGUF) | GGUF and MLX from Hugging Face |
| Options | Every llama.cpp flag | A curated subset | Most, in menus |
| New models | Usually first | After Ollama adds them | After an app update |
| Best for | Control, servers, latest models | Easiest setup | Point-and-click users |
If you want a full comparison with speed numbers, see vLLM vs llama.cpp vs Ollama.
Install llama-server
llama-server comes with every llama.cpp install, next to llama-cli. Pick one method:
| Method | System | Command |
|---|---|---|
| Winget | Windows | winget install llama.cpp |
| Homebrew | macOS, Linux | brew install llama.cpp |
| Install script | macOS, Linux | curl -LsSf https://llama.app/install.sh | sh |
| Install script | Windows PowerShell | irm https://llama.app/install.ps1 | iex |
| Prebuilt zip | All | GitHub Releases page of ggml-org/llama.cpp |
NVIDIA on Windows: if you download a zip, pick a CUDA build (and the matching cudart zip if the release lists one), not the CPU build, or nothing will run on your GPU. AMD users can pick the Vulkan or ROCm build. Unzip it to a folder such as C:\llama and run commands from there.
Build from source if you want the very latest code:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -t llama-server
The binary ends up in build/bin/. Leave out -DGGML_CUDA=ON on a Mac; Metal is enabled by default there.
Check it works:
llama-server --version
Start Your First Server
The quickest start downloads a model from Hugging Face for you:
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080
-hf takes a repository name and an optional quant after a colon, for example unsloth/Qwen3-8B-GGUF:Q4_K_M. Without a quant it picks Q4_K_M, or the first file if there is no Q4_K_M. If the model has a vision projector (mmproj), that is downloaded too. Files are cached, so the next start is instant.
If you already have a GGUF file, point to it with -m:
llama-server -m D:\models\Qwen3-8B-Q4_K_M.gguf --port 8080
Wait for the log to say the server is listening, then open http://127.0.0.1:8080 in your browser. You get Llama Chat, the built-in chat app, with conversation history, file and image upload (for vision models) and settings for temperature and other options.
Need models? My guide to downloading models from Hugging Face explains how to pick the right GGUF quant.
The Flags That Matter
llama-server has well over a hundred options. These are the ones most people need:
| Flag | What it does | Default |
|---|---|---|
-m FILE |
Load a local GGUF file | none |
-hf user/repo[:quant] |
Download and load from Hugging Face | none |
--port N |
Port to listen on | 9931 |
--host IP |
Address to listen on (0.0.0.0 for your network) |
127.0.0.1 |
-c N |
Context size in tokens | 0 = from the model (with –fit adjusting it) |
-ngl N |
Layers on the GPU: a number, auto or all |
auto |
-fa on|off|auto |
Flash Attention | auto |
-ctk / -ctv TYPE |
KV cache type, e.g. q8_0 to save memory |
f16 |
-np N |
Number of parallel slots (users) | auto |
--fit on|off |
Adjust unset options to fit in memory | on |
--n-cpu-moe N |
Keep MoE expert weights of the first N layers on the CPU | off |
--api-key KEY |
Require a key for API calls | none |
-a NAME |
Model name reported to the API | file name |
--jinja |
Use the model’s chat template (needed for tool calls) | on |
--reasoning-budget N |
Limit thinking tokens (0 = off, -1 = no limit) | -1 |
Most options can also be set with environment variables (for example LLAMA_ARG_CTX_SIZE), which is handy in Docker and services. Run llama-server --help for the full list.
GPU Layers, Context and Memory
Older guides tell you to always pass -ngl 99 and a fixed -c. You no longer have to.
With --fit on (the default), llama-server looks at your free VRAM and sets the options you did not set yourself: how many layers go on the GPU and how big the context can be.
It aims to leave about 1 GB free per GPU (change it with --fit-target) and will not pick a context below 4,096 tokens (--fit-ctx).
Set them yourself when you want control:
- Force everything onto the GPU:
-ngl all. If it does not fit, loading fails or spills. - Pin the context:
-c 32768. Useful when you run other GPU apps and want to leave room. - Save memory on context:
-ctk q8_0 -ctv q8_0roughly halves the KV cache. - Big MoE models on a small GPU:
--n-cpu-moe 20(or--cpu-moefor all layers) keeps the large expert weights in system RAM while attention runs on the GPU. This is how people run 100B+ MoE models on 16-24 GB cards with lots of RAM. - Two GPUs: llama-server splits layers across both by default. Use
-ts 3,1to change the split, or-dev CUDA0to use one card only.
Example for a 16 GB card running a 14B model with a 32k context:
llama-server -m Qwen3-14B-Q4_K_M.gguf -c 32768 -ngl all -fa on -ctk q8_0 -ctv q8_0 --port 8080
For model-specific settings, see running Qwen3.8-27B on a 16 GB GPU and my NVFP4 vs MXFP4 guide for RTX 50 cards.
Use the OpenAI-Compatible API
llama-server speaks the OpenAI Chat Completions, Responses and Embeddings formats, plus the Anthropic Messages format. Any OpenAI client works by changing the base URL.
curl:
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"Explain GGUF in one paragraph"}]}'
Python (openai package):
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
r = client.chat.completions.create(
model="local",
messages=[{"role": "user", "content": "Hello!"}])
print(r.choices[0].message.content)
With one model loaded, the model name does not matter. Use -a to set a friendly name if your app shows it.
Useful extras:
- Health check:
GET /healthreturns{"status": "ok"}when ready, or 503 while the model is still loading. - JSON output: send
response_formatwith a JSON schema to force structured answers. - Thinking control: send
"chat_template_kwargs": {"enable_thinking": false}to turn off reasoning for models that support it. - Embeddings: start an embedding model with
--embeddingand call/v1/embeddings.
Connect Open WebUI, Agents and Other Apps
Any app with an “OpenAI-compatible” or “custom OpenAI” provider can use llama-server:
- Base URL:
http://127.0.0.1:8080/v1 - API key: anything (or the key you set with
--api-key) - Model: the alias you set, or any name
In Open WebUI, add it under Settings, Connections, OpenAI API. My Open WebUI setup guide shows the screens. The same settings work for Aider, Cursor with local models, SillyTavern and AnythingLLM.
For agents and tool calling, keep --jinja on (it is the default) so the model’s own chat template handles tool calls, and give it a bigger context (32k or more).
Several Users and Several Models
Several users on one model. -np 4 creates four slots, so four requests run at the same time with continuous batching. The context is shared out between slots, so raise -c to match: -np 4 -c 32768 gives each slot about 8k tokens.
Several models (router mode). Start llama-server without a model and point it at a folder:
llama-server --models-dir D:\models --port 8080
The server now lists every GGUF in that folder and loads the one named in each request’s model field, unloading the least recently used one when needed (up to 4 at once by default, change with --models-max).
Models with several files, such as a vision projector or split shards, go in their own subfolder. Arguments you pass to the router, such as -c 8192, apply to every model. This gives you an Ollama-style “pick any model” setup with plain GGUF files.
Network Access and API Keys
By default llama-server only listens on 127.0.0.1, so only your own PC can reach it. To use it from your phone or another PC on your network:
llama-server -m model.gguf --host 0.0.0.0 --port 8080 --api-key YOUR_LONG_RANDOM_KEY
- Allow the port through your firewall, and connect with your PC’s local IP (for example
http://192.168.1.20:8080). - Always set
--api-keywhen listening on the network. Clients then send it asAuthorization: Bearer YOUR_KEY. - Do not open the port to the internet directly. Use a VPN such as Tailscale or a reverse proxy with HTTPS.
Run It in Docker
Official images are published as ghcr.io/ggml-org/llama.cpp:server (CPU) and :server-cuda (NVIDIA):
docker run -p 8080:8080 -v /path/to/models:/models --gpus all \
ghcr.io/ggml-org/llama.cpp:server-cuda \
-m /models/Qwen3-8B-Q4_K_M.gguf -c 8192 --host 0.0.0.0 --port 8080 -ngl 99
The container needs --host 0.0.0.0 so the port mapping works, and the NVIDIA Container Toolkit for --gpus all. On Windows, Docker Desktop with WSL2 handles this; see my WSL2 local AI setup guide.
Troubleshooting
- “Connection refused” from another app. Check the port. The default is 9931 now, and many apps assume 8080. Pass
--portexplicitly and use the same number everywhere. - It runs on the CPU and is slow. You installed a CPU build. Get the CUDA (or Vulkan/ROCm) build, then check the start-up log lists your GPU. Run
llama-server --list-devicesto see what it detects. - Out of memory while loading. Lower
-c, use-ctk q8_0 -ctv q8_0, pick a smaller quant, or remove a manual-ngl alland let--fitdecide. - Garbled answers or the model never stops. The chat template is wrong or missing. Keep
--jinjaon, update llama.cpp, and download the GGUF again from a maintained repo. - Tool calls do not work. Use a model with tool-call support in its template, keep
--jinjaon, and update to the latest release. - 503 “Loading model”. Normal while a large model loads. Wait until
/healthreturns ok. - “unknown model architecture”. Your build is too old for that model. Update llama.cpp.
FAQ
Is llama-server faster than Ollama?
Often slightly, because you can tune every option and get new optimizations first. With the same settings, the difference is usually small.
Can I use my Ollama models with llama-server?
Ollama stores GGUF files as blobs with hashed names, so it is easier to download the same model as a normal GGUF. You can point -m at an Ollama blob, but the file name will not tell you which model it is.
Does llama-server support vision models?
Yes. Use -hf to fetch the model and its mmproj file together, or pass --mmproj with a local file. Images then work in the web chat and the API.
How do I run it in the background on Windows?
Make a .bat file with your command and add a shortcut to it in the Startup folder, or run it as a service with a tool such as NSSM.
Can I turn off the web chat?
Yes. Run with --no-webui if you only want the API.
Sources
- llama.cpp server README (all flags, defaults, endpoints, router mode, Docker)
- llama.cpp GitHub Releases (prebuilt builds)
- llama.cpp on GitHub (build instructions)



