How to Run Ollama in Docker With GPU (NVIDIA, AMD) + Open WebUI Compose (2026)

Running Ollama in Docker keeps your system clean, makes updates a one-line job and lets you move the whole setup to another machine or a cloud GPU in minutes. It is also the easiest way to run Ollama next to Open WebUI, n8n or your own app.

This guide shows you how to run Ollama in Docker step by step: CPU only, NVIDIA GPU, AMD GPU and Vulkan, plus a ready Docker Compose file with Open WebUI, the settings worth changing, how to update safely and how to fix the errors people hit most.

Why Run Ollama in Docker

Ollama already has simple installers for Windows, macOS and Linux (see how to install Ollama). Docker is worth it when you want one of these:

• A clean host: no service files, libraries or users added to your system. Delete the container and it is gone.
• Easy updates and rollbacks: pull a new image tag, or go back to the old one if a release breaks something.
• Stacks: run Ollama, Open WebUI, n8n and a database together with one Compose file.
• Servers and cloud GPUs: the same command works on a home server, a rented GPU box or a Kubernetes node.
• Several versions side by side: test a new Ollama build on another port without touching your main one.

The trade-off is one extra layer. GPU access needs the right container runtime, and you run commands through docker exec. Speed on the GPU is the same as a native install once it is set up.

Which Setup Do You Need

Ollama’s hardware page lists NVIDIA cards with compute capability 5.0 or newer and driver 550 or newer (cards with compute capability 5.0 to 6.2 need driver 570+). That covers everything from a GTX 750 Ti to the RTX 50 series.

For AMD on Linux it lists cards such as the RX 9070 XT, RX 7900 XTX, RX 6800 and Ryzen AI Max+ 395.

Before You Start

• Docker installed: Docker Engine on Linux, Docker Desktop on Windows. Check with docker --version.
• A GPU driver on the host: the container brings its own CUDA libraries, but the driver lives on the host. On NVIDIA run nvidia-smi; it must list your card.
• Disk space: the image is a few GB and models are bigger. A 4B model is about 3 GB, a 27B to 32B model at 4-bit is roughly 17 to 20 GB.
• Enough VRAM: check what fits with how much VRAM you need for AI.

Step 1: Run Ollama in Docker (CPU)

This works on any machine and is a good first test even if you have a GPU.

1. Start the container.

   docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

   -d runs it in the background, -v ollama:/root/.ollama keeps your models in a named volume so they survive updates, and -p 11434:11434 opens the API port.

2. Check that it is running.

   docker ps
   curl http://localhost:11434

   The curl call should answer Ollama is running.

3. Run a small model.

   docker exec -it ollama ollama run llama3.2

   The first run downloads the model. Type a question, and /bye to leave.

That is a full working Ollama server. Any app on your PC can now use http://localhost:11434, including the OpenAI-compatible endpoint at http://localhost:11434/v1.

Step 2: Add an NVIDIA GPU

Docker cannot see your GPU until the NVIDIA Container Toolkit is installed. These are the commands from Ollama’s Docker docs for Ubuntu and Debian.

1. Add NVIDIA’s repository.

   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
       | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
   curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
       | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
       | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
   sudo apt-get update

   On Fedora, RHEL or Rocky use the yum repo instead:

   curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
       | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

2. Install the toolkit.

   sudo apt-get install -y nvidia-container-toolkit
   # or: sudo yum install -y nvidia-container-toolkit

3. Connect it to Docker and restart Docker.

   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker

4. Test GPU access. This is the check Ollama’s troubleshooting page recommends:

   docker run --rm --gpus all ubuntu nvidia-smi

   If you see your GPU table, the runtime works. If not, fix this first; Ollama will not see the GPU either.

5. Recreate the Ollama container with the GPU.

   docker rm -f ollama
   docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

   Your models stay in the ollama volume, so nothing is downloaded again.

6. Confirm the model is on the GPU.

   docker exec -it ollama ollama run gemma4 "Say hi"
   docker exec ollama ollama ps

   The PROCESSOR column should say 100% GPU. A split such as 48%/52% CPU/GPU means the model or context is too big for your VRAM.

On an NVIDIA Jetson, Ollama cannot detect the JetPack version by itself. Add -e JETSON_JETPACK=5 or -e JETSON_JETPACK=6.

AMD GPU and Vulkan

For AMD Radeon cards on Linux, use the ROCm image and pass the GPU devices into the container:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

The host needs AMD’s ROCm v7 driver (install it with amdgpu-install from AMD’s ROCm docs). Ollama also recommends AMD’s latest Linux driver over the older one built into the kernel.

If your Radeon card is not on ROCm’s list, you can make ROCm treat it as a close supported chip. For example, an RX 5400 (gfx1034) can try:

-e HSA_OVERRIDE_GFX_VERSION=10.3.0

Vulkan is now bundled in the normal ollama/ollama image and turns on by default when the container can reach the GPU devices. It is the fallback for GPUs that ROCm does not support:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Set -e OLLAMA_VULKAN=0 to switch Vulkan off, or -e GGML_VK_VISIBLE_DEVICES=0 to choose a device.

Ollama in Docker on Windows (WSL2)

Ollama’s FAQ says GPU acceleration in Docker works on Linux and on Windows with WSL2. On Windows:

1. Install the latest NVIDIA Game Ready or Studio driver in Windows. Do not install a Linux driver inside WSL; WSL uses the Windows driver.
2. Install Docker Desktop and make sure Settings > General > “Use the WSL 2 based engine” is on.
3. Test the GPU in PowerShell: docker run --rm --gpus all ubuntu nvidia-smi
4. Start Ollama with the same command as Linux:

   docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

If model downloads crawl inside WSL2 on Windows 10, Ollama’s FAQ suggests disabling Large Send Offload Version 2 (IPv4 and IPv6) on the vEthernet (WSL) network adapter.

Honestly, for a single Windows gaming PC the native Ollama app is simpler. Docker on Windows makes sense when you also run other containers such as Open WebUI.

Pull and Run Models

Every normal Ollama command works through docker exec. The full list is in our Ollama commands cheat sheet.

Tired of typing docker exec? Add an alias on Linux: alias ollama='docker exec -it ollama ollama'. Not sure which model to pull? Start with our list of the best Ollama models.

You can also call the API from your host:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{"role": "user", "content": "Explain Docker volumes in one line"}]
}'

Docker Compose With Open WebUI

Most people want a ChatGPT-style interface on top. This Compose file is based on Open WebUI’s official one, with the NVIDIA GPU block from their GPU file added. Save it as docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    tty: true
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=30m
      - OLLAMA_CONTEXT_LENGTH=16384
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama: {}
  open-webui: {}

1. Start the stack: docker compose up -d
2. Open http://localhost:3000 and create the first account. The first account becomes the admin.
3. Pull a model from Open WebUI’s model picker, or with docker exec ollama ollama pull gemma4.

Notes on this file:

• No GPU? Delete the whole deploy: block.
• AMD? Change the image to ollama/ollama:rocm, delete deploy: and add devices: ["/dev/kfd", "/dev/dri"] under the ollama service.
• Containers talk by name: Open WebUI reaches Ollama at http://ollama:11434, not localhost.
• Port 11434 is bound to 127.0.0.1 so only your PC can reach the raw API. Remove 127.0.0.1: only if you need it on your network and understand the risk below.

Want more from Open WebUI, such as RAG over your documents or web search? See our Open WebUI with Ollama setup guide. To build automations on the same stack, add n8n with our n8n and Ollama local AI agent guide.

Useful Environment Variables

Pass these with -e NAME=value in docker run, or under environment: in Compose. The names and defaults come from Ollama’s own config code.

A good GPU starting point for a 16 GB card running a mid-size model:

docker run -d --gpus=all \
  -e OLLAMA_CONTEXT_LENGTH=16384 \
  -e OLLAMA_KV_CACHE_TYPE=q8_0 \
  -e OLLAMA_FLASH_ATTENTION=1 \
  -e OLLAMA_KEEP_ALIVE=30m \
  -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 \
  --restart unless-stopped --name ollama ollama/ollama

Check the real context and placement with docker exec ollama ollama ps; it shows a CONTEXT column next to PROCESSOR.

Update, Back Up and Remove

Update Ollama (models are kept because they live in the volume):

docker pull ollama/ollama
docker rm -f ollama
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

With Compose it is just docker compose pull then docker compose up -d.

Pin a version when you need stability: use a tag such as ollama/ollama:0.x.y (the exact numbers are on the Ollama releases page) instead of latest, and change it on purpose.

Back up your models by archiving the volume:

docker run --rm -v ollama:/data -v "$PWD":/backup ubuntu tar czf /backup/ollama-models.tgz -C /data .

Remove everything:

docker rm -f ollama
docker volume rm ollama      # deletes all downloaded models
docker image rm ollama/ollama

Removing a native install is a different process; see how to delete Ollama models and uninstall Ollama.

Keep Your Ollama Container Safe

The Ollama image listens on all interfaces inside the container, and the plain API has no password. So the -p flag decides who can reach it.

• Use -p 127.0.0.1:11434:11434 on any machine with a public IP, such as a cloud GPU. Open ports are scanned constantly, and an open Ollama lets strangers use your GPU and pull or delete models.
• Need remote access? Put Open WebUI (which has logins) or a reverse proxy with authentication in front, or use a private network such as Tailscale or a VPN.
• Docker skips UFW: on Ubuntu, published Docker ports can bypass UFW rules. Binding to 127.0.0.1 is the reliable fix.
• Fully local: add -e OLLAMA_NO_CLOUD=1 if you do not want cloud models or web search available.

Troubleshooting

“could not select device driver “” with capabilities: [[gpu]]”

Docker does not know about the NVIDIA runtime. Install the NVIDIA Container Toolkit, run sudo nvidia-ctk runtime configure --runtime=docker and sudo systemctl restart docker, then retry.

Ollama runs on the CPU even with –gpus=all

• Run docker run --rm --gpus all ubuntu nvidia-smi. If that fails, the problem is the host driver or toolkit, not Ollama.
• Check the driver is 550 or newer with nvidia-smi.
• Read the logs: docker logs ollama. Restart with -e OLLAMA_DEBUG=1 for GPU discovery details.
• If ollama ps shows a CPU/GPU split, the model plus context does not fit. Use a smaller model or quant, lower OLLAMA_CONTEXT_LENGTH or set OLLAMA_KV_CACHE_TYPE=q8_0. Our GPU VRAM usage guide shows how to watch memory live.

GPU works at first, then drops to CPU after a while

This is a known Linux Docker issue. Ollama’s fix: edit /etc/docker/daemon.json, add "exec-opts": ["native.cgroupdriver=cgroupfs"], then restart Docker.

GPU lost after sleep or resume (Linux)

Reload the NVIDIA UVM driver on the host: sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm, then restart the container.

Error codes 3, 46, 100 or 999 in the logs

These are GPU initialization errors. Ollama suggests sudo nvidia-modprobe -u, reloading nvidia_uvm, a reboot and the latest driver, in that order.

Open WebUI shows no models

Inside a container, localhost means the container itself. Set OLLAMA_BASE_URL=http://ollama:11434 when both are in one Compose file, or http://host.docker.internal:11434 when Ollama runs on the host (on Linux add --add-host=host.docker.internal:host-gateway).

“bind: address already in use” on port 11434

A native Ollama is already running. Quit it (or sudo systemctl stop ollama on Linux), or map the container to another port with -p 11435:11434.

503 errors under load

The queue is full. Raise OLLAMA_MAX_QUEUE or OLLAMA_NUM_PARALLEL if you have spare VRAM, or add a second container on another GPU.

Out of memory or the container restarts

Lower the context, use a smaller quant or stop other models with ollama stop. Our CUDA out of memory fix guide covers the general steps.

My Take

For a single desktop, the native app is fine. As soon as you run a second tool next to Ollama, or you work on a Linux server or a rented GPU, Docker is the cleaner choice.

Set up the NVIDIA Container Toolkit once, keep models in a named volume, bind the port to 127.0.0.1, and updates become a two-minute job.

The Compose file above is the setup I would hand a friend: Ollama on the GPU, Open WebUI on port 3000, a sensible context size and nothing exposed to the internet. Still deciding if Ollama is the right engine? Compare it in Ollama vs LM Studio vs llama.cpp.

FAQ

What is the Docker command to run Ollama with a GPU?

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. It needs the NVIDIA Container Toolkit on the host first.

Does Ollama in Docker work on Mac?

It runs, but only on the CPU. Docker Desktop on macOS has no GPU passthrough, so the native Ollama app is much faster on Apple Silicon.

Is Ollama slower in Docker?

No, not on Linux with the GPU set up correctly. The model runs on the same GPU through the same CUDA or ROCm libraries. Loading from a Windows-mounted folder can be slower, so keep models in a Docker volume.

Where does Ollama store models in Docker?

In /root/.ollama/models inside the container, which the -v ollama:/root/.ollama flag maps to a named Docker volume called ollama.

How do I update Ollama in Docker without losing models?

Run docker pull ollama/ollama, remove the old container and start a new one with the same -v ollama:/root/.ollama volume. Models stay in the volume.

Which image do I use for an AMD GPU?

ollama/ollama:rocm with --device /dev/kfd --device /dev/dri on Linux. The standard image also has Vulkan built in for GPUs that ROCm does not support.

How do I connect another container to Ollama?

Put both in the same Compose file or Docker network and use http://ollama:11434. For the OpenAI-compatible API, use http://ollama:11434/v1.

How do I increase the context length in Docker?

Add -e OLLAMA_CONTEXT_LENGTH=32768 (or any value your VRAM allows) and check the result with ollama ps.

Sources

• Ollama docs: Docker
• Ollama docs: Hardware support
• Ollama docs: FAQ and Troubleshooting
• Ollama docs: Context length
• Ollama source: environment variables and defaults
• ollama/ollama on Docker Hub
• NVIDIA Container Toolkit install guide
• Open WebUI official docker-compose.yaml