Skip to content
,

llama-server Guide (2026): Run llama.cpp as a Local OpenAI-Compatible API

8–13 minutes
Woman in a home lab with a PC tower holding a card reading SERVER, hook Your own local AI API

llama-server is the server that ships with llama.cpp. One command turns any GGUF model into a local chat app in your browser plus an OpenAI-compatible API that Open WebUI, coding agents, LangChain and your own scripts can use.

It is lighter than Ollama, gives you every llama.cpp option directly, and usually gets new model support first.

This guide covers installing it on Windows, Mac and Linux, the commands you actually need, the flags that matter for speed and memory, calling the API, running several users or models, and fixing the common problems. Every flag and default here is checked against the current llama.cpp server documentation.

TL;DR

  • Install: winget install llama.cpp (Windows) or brew install llama.cpp (Mac/Linux), or a prebuilt zip from GitHub Releases.
  • Run: llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080 downloads a model and starts the server.
  • Open: http://127.0.0.1:8080 for the chat app. The API is at /v1/chat/completions.
  • Default port is now 9931, not 8080. Always pass --port so other tools know where to connect.
  • Memory is automatic: --fit (on by default) picks GPU layers and context to fit your VRAM. Set -c and -ngl yourself only when you want control.

New to running models locally? My Ollama guide is the easiest starting point, and my llama.cpp 1.0 guide explains what changed in the latest release.

llama-server vs Ollama vs LM Studio

llama-server Ollama LM Studio
Engine llama.cpp itself Its own engine plus llama.cpp llama.cpp and MLX
Interface Command line + built-in web chat App + CLI Desktop app
Model files Any GGUF file or Hugging Face repo Ollama library (or imported GGUF) GGUF and MLX from Hugging Face
Options Every llama.cpp flag A curated subset Most, in menus
New models Usually first After Ollama adds them After an app update
Best for Control, servers, latest models Easiest setup Point-and-click users

If you want a full comparison with speed numbers, see vLLM vs llama.cpp vs Ollama.

Install llama-server

llama-server comes with every llama.cpp install, next to llama-cli. Pick one method:

Method System Command
Winget Windows winget install llama.cpp
Homebrew macOS, Linux brew install llama.cpp
Install script macOS, Linux curl -LsSf https://llama.app/install.sh | sh
Install script Windows PowerShell irm https://llama.app/install.ps1 | iex
Prebuilt zip All GitHub Releases page of ggml-org/llama.cpp

NVIDIA on Windows: if you download a zip, pick a CUDA build (and the matching cudart zip if the release lists one), not the CPU build, or nothing will run on your GPU. AMD users can pick the Vulkan or ROCm build. Unzip it to a folder such as C:\llama and run commands from there.

Build from source if you want the very latest code:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -t llama-server

The binary ends up in build/bin/. Leave out -DGGML_CUDA=ON on a Mac; Metal is enabled by default there.

Check it works:

llama-server --version

Start Your First Server

The quickest start downloads a model from Hugging Face for you:

llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080

-hf takes a repository name and an optional quant after a colon, for example unsloth/Qwen3-8B-GGUF:Q4_K_M. Without a quant it picks Q4_K_M, or the first file if there is no Q4_K_M. If the model has a vision projector (mmproj), that is downloaded too. Files are cached, so the next start is instant.

If you already have a GGUF file, point to it with -m:

llama-server -m D:\models\Qwen3-8B-Q4_K_M.gguf --port 8080

Wait for the log to say the server is listening, then open http://127.0.0.1:8080 in your browser. You get Llama Chat, the built-in chat app, with conversation history, file and image upload (for vision models) and settings for temperature and other options.

Need models? My guide to downloading models from Hugging Face explains how to pick the right GGUF quant.

The Flags That Matter

llama-server has well over a hundred options. These are the ones most people need:

Flag What it does Default
-m FILE Load a local GGUF file none
-hf user/repo[:quant] Download and load from Hugging Face none
--port N Port to listen on 9931
--host IP Address to listen on (0.0.0.0 for your network) 127.0.0.1
-c N Context size in tokens 0 = from the model (with –fit adjusting it)
-ngl N Layers on the GPU: a number, auto or all auto
-fa on|off|auto Flash Attention auto
-ctk / -ctv TYPE KV cache type, e.g. q8_0 to save memory f16
-np N Number of parallel slots (users) auto
--fit on|off Adjust unset options to fit in memory on
--n-cpu-moe N Keep MoE expert weights of the first N layers on the CPU off
--api-key KEY Require a key for API calls none
-a NAME Model name reported to the API file name
--jinja Use the model’s chat template (needed for tool calls) on
--reasoning-budget N Limit thinking tokens (0 = off, -1 = no limit) -1

Most options can also be set with environment variables (for example LLAMA_ARG_CTX_SIZE), which is handy in Docker and services. Run llama-server --help for the full list.

GPU Layers, Context and Memory

Older guides tell you to always pass -ngl 99 and a fixed -c. You no longer have to.

With --fit on (the default), llama-server looks at your free VRAM and sets the options you did not set yourself: how many layers go on the GPU and how big the context can be.

It aims to leave about 1 GB free per GPU (change it with --fit-target) and will not pick a context below 4,096 tokens (--fit-ctx).

Set them yourself when you want control:

  • Force everything onto the GPU: -ngl all. If it does not fit, loading fails or spills.
  • Pin the context: -c 32768. Useful when you run other GPU apps and want to leave room.
  • Save memory on context: -ctk q8_0 -ctv q8_0 roughly halves the KV cache.
  • Big MoE models on a small GPU: --n-cpu-moe 20 (or --cpu-moe for all layers) keeps the large expert weights in system RAM while attention runs on the GPU. This is how people run 100B+ MoE models on 16-24 GB cards with lots of RAM.
  • Two GPUs: llama-server splits layers across both by default. Use -ts 3,1 to change the split, or -dev CUDA0 to use one card only.

Example for a 16 GB card running a 14B model with a 32k context:

llama-server -m Qwen3-14B-Q4_K_M.gguf -c 32768 -ngl all -fa on -ctk q8_0 -ctv q8_0 --port 8080

For model-specific settings, see running Qwen3.8-27B on a 16 GB GPU and my NVFP4 vs MXFP4 guide for RTX 50 cards.

Use the OpenAI-Compatible API

llama-server speaks the OpenAI Chat Completions, Responses and Embeddings formats, plus the Anthropic Messages format. Any OpenAI client works by changing the base URL.

curl:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Explain GGUF in one paragraph"}]}'

Python (openai package):

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello!"}])
print(r.choices[0].message.content)

With one model loaded, the model name does not matter. Use -a to set a friendly name if your app shows it.

Useful extras:

  • Health check: GET /health returns {"status": "ok"} when ready, or 503 while the model is still loading.
  • JSON output: send response_format with a JSON schema to force structured answers.
  • Thinking control: send "chat_template_kwargs": {"enable_thinking": false} to turn off reasoning for models that support it.
  • Embeddings: start an embedding model with --embedding and call /v1/embeddings.

Connect Open WebUI, Agents and Other Apps

Any app with an “OpenAI-compatible” or “custom OpenAI” provider can use llama-server:

  • Base URL: http://127.0.0.1:8080/v1
  • API key: anything (or the key you set with --api-key)
  • Model: the alias you set, or any name

In Open WebUI, add it under Settings, Connections, OpenAI API. My Open WebUI setup guide shows the screens. The same settings work for Aider, Cursor with local models, SillyTavern and AnythingLLM.

For agents and tool calling, keep --jinja on (it is the default) so the model’s own chat template handles tool calls, and give it a bigger context (32k or more).

Several Users and Several Models

Several users on one model. -np 4 creates four slots, so four requests run at the same time with continuous batching. The context is shared out between slots, so raise -c to match: -np 4 -c 32768 gives each slot about 8k tokens.

Several models (router mode). Start llama-server without a model and point it at a folder:

llama-server --models-dir D:\models --port 8080

The server now lists every GGUF in that folder and loads the one named in each request’s model field, unloading the least recently used one when needed (up to 4 at once by default, change with --models-max).

Models with several files, such as a vision projector or split shards, go in their own subfolder. Arguments you pass to the router, such as -c 8192, apply to every model. This gives you an Ollama-style “pick any model” setup with plain GGUF files.

Network Access and API Keys

By default llama-server only listens on 127.0.0.1, so only your own PC can reach it. To use it from your phone or another PC on your network:

llama-server -m model.gguf --host 0.0.0.0 --port 8080 --api-key YOUR_LONG_RANDOM_KEY
  • Allow the port through your firewall, and connect with your PC’s local IP (for example http://192.168.1.20:8080).
  • Always set --api-key when listening on the network. Clients then send it as Authorization: Bearer YOUR_KEY.
  • Do not open the port to the internet directly. Use a VPN such as Tailscale or a reverse proxy with HTTPS.

Run It in Docker

Official images are published as ghcr.io/ggml-org/llama.cpp:server (CPU) and :server-cuda (NVIDIA):

docker run -p 8080:8080 -v /path/to/models:/models --gpus all \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-8B-Q4_K_M.gguf -c 8192 --host 0.0.0.0 --port 8080 -ngl 99

The container needs --host 0.0.0.0 so the port mapping works, and the NVIDIA Container Toolkit for --gpus all. On Windows, Docker Desktop with WSL2 handles this; see my WSL2 local AI setup guide.

Troubleshooting

  • “Connection refused” from another app. Check the port. The default is 9931 now, and many apps assume 8080. Pass --port explicitly and use the same number everywhere.
  • It runs on the CPU and is slow. You installed a CPU build. Get the CUDA (or Vulkan/ROCm) build, then check the start-up log lists your GPU. Run llama-server --list-devices to see what it detects.
  • Out of memory while loading. Lower -c, use -ctk q8_0 -ctv q8_0, pick a smaller quant, or remove a manual -ngl all and let --fit decide.
  • Garbled answers or the model never stops. The chat template is wrong or missing. Keep --jinja on, update llama.cpp, and download the GGUF again from a maintained repo.
  • Tool calls do not work. Use a model with tool-call support in its template, keep --jinja on, and update to the latest release.
  • 503 “Loading model”. Normal while a large model loads. Wait until /health returns ok.
  • “unknown model architecture”. Your build is too old for that model. Update llama.cpp.

FAQ

Is llama-server faster than Ollama?

Often slightly, because you can tune every option and get new optimizations first. With the same settings, the difference is usually small.

Can I use my Ollama models with llama-server?

Ollama stores GGUF files as blobs with hashed names, so it is easier to download the same model as a normal GGUF. You can point -m at an Ollama blob, but the file name will not tell you which model it is.

Does llama-server support vision models?

Yes. Use -hf to fetch the model and its mmproj file together, or pass --mmproj with a local file. Images then work in the web chat and the API.

How do I run it in the background on Windows?

Make a .bat file with your command and add a shortcut to it in the Startup folder, or run it as a service with a tool such as NSSM.

Can I turn off the web chat?

Yes. Run with --no-webui if you only want the API.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Woman in a home lab with a PC tower holding a card reading SERVER, hook Your own local AI API