Skip to content
,

How to Run an LLM on Raspberry Pi 5: Ollama, llama.cpp and Best Models (2026)

8–12 minutes
Surprised woman holding a PI 5 card with the text Offline AI on a Pi 5, featured image for running an LLM on Raspberry Pi 5

Yes, you can run a real AI chatbot on a Raspberry Pi. A Raspberry Pi 5 can run small language models fully offline, with no cloud, no subscription and only a few watts of power.

It will not match ChatGPT, and big models are far too slow. But for a home assistant, a private offline helper, a learning project or a smart device, a Pi running a small model is fun and genuinely useful.

This guide shows exactly how to do it: what hardware to buy, how to set up the OS, how to run models with Ollama or llama.cpp, which models work best, how to add a web chat UI, and how to fix the common problems.

Hardware details come from Raspberry Pi’s product pages, and setup commands from the official Ollama and llama.cpp docs, checked in November 2026.

TL;DR

  • Board: Raspberry Pi 5 with 8 GB or 16 GB RAM. 4 GB only handles tiny models. Older Pis are too slow.
  • Must have: the official 27W USB-C power supply, the Active Cooler, and a 64-bit OS. An NVMe SSD (via an M.2 HAT) makes model loading much faster.
  • Easiest: curl -fsSL https://ollama.com/install.sh | sh, then ollama run a 1B to 4B model.
  • Fastest: build llama.cpp and run Q4 GGUF models.
  • Best models: 0.5B to 4B instruct models at Q4. 7B to 8B works on 8 GB+ boards but is slow.
  • Want more speed? The $130 AI HAT+ 2 (Hailo-10H, 40 TOPS, 8 GB on board) runs selected small LLMs.
  • Price note: RAM shortages pushed the Pi 5 16 GB to about $385 in November 2026.

What Can a Raspberry Pi Really Do?

A Raspberry Pi 5 has a quad-core Arm Cortex-A76 CPU at 2.4 GHz and up to 16 GB of LPDDR4X memory. LLMs on a Pi run on the CPU, so speed depends on model size and memory bandwidth.

Model size (Q4) RAM used (approx.) How it feels on a Pi 5 Good for
0.5B to 1.5B 0.5 to 1.5 GB Quick, chat feels responsive Simple Q&A, commands, home automation
3B to 4B 2 to 3 GB Usable, reads at a slow typing pace General chat, summaries, writing help
7B to 8B 4.5 to 6 GB Slow, you wait for each sentence Better answers when speed does not matter
13B+ 8 GB+ Very slow Not recommended

The rule is simple: smaller model, faster replies. A model file must also fit in RAM with room left for the OS, so a 4 GB Pi tops out around 1B to 3B models.

Hardware You Need

Part Recommendation Why
Board Raspberry Pi 5, 8 GB or 16 GB More RAM = bigger models; Pi 4 and older are much slower
Power Official 27W USB-C supply (5V/5A) Weak supplies cause throttling and crashes under load
Cooling Active Cooler or the official case with fan AI keeps all 4 cores at 100%; without cooling it throttles
Storage NVMe SSD via an M.2 HAT (PCIe 2.0 x1), or a fast A2 microSD SSD loads models much faster than microSD
OS Raspberry Pi OS 64-bit (Lite is fine) or Ubuntu 64-bit Ollama and llama.cpp need 64-bit Arm
Optional AI HAT+ 2 ($130) Offloads supported small LLMs to a 40 TOPS NPU

About prices: memory shortages have pushed Pi prices up several times in 2026. The 16 GB Pi 5 rose to about $385 in November 2026, so check current reseller prices before buying. For many people, an 8 GB board is the sweet spot for small models.

Step 1: Install a 64-bit OS

  1. Download Raspberry Pi Imager on your PC from raspberrypi.com.
  2. Choose your device (Raspberry Pi 5) and Raspberry Pi OS (64-bit). Pick Lite if you will only use the terminal; it saves RAM.
  3. Open the settings in Imager to set a username, Wi-Fi and enable SSH.
  4. Write it to your microSD card or NVMe drive and boot the Pi.
  5. Update everything: sudo apt update && sudo apt full-upgrade -y, then reboot.
  6. Confirm 64-bit: uname -m must print aarch64.

Step 2: Run Models With Ollama

Ollama is the easiest way to start. It supports 64-bit Arm Linux, downloads models for you and gives you an API.

  1. Install Ollama: curl -fsSL https://ollama.com/install.sh | sh
  2. Check it runs: ollama -v
  3. Run a small model: ollama run llama3.2:1b. The first run downloads it.
  4. Try a slightly bigger one if you have 8 GB: ollama run llama3.2 (3B) or a similar 3B to 4B model.
  5. See memory use: ollama ps while a model is loaded. On a Pi it will show 100% CPU, which is normal.
  6. Use the API from other devices at http://<pi-ip>:11434 after setting OLLAMA_HOST=0.0.0.0 in the service.

New to Ollama? Our complete Ollama guide covers every command, Modelfiles and the API.

Step 3: Run Models With llama.cpp (Faster)

llama.cpp is the engine behind many local AI apps. Building it on the Pi gives you the most control and often the best speed.

  1. Install build tools: sudo apt install -y git cmake build-essential libcurl4-openssl-dev
  2. Get the code: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
  3. Build it: cmake -B build then cmake --build build --config Release -j 4. This takes a while on a Pi.
  4. Download a GGUF model (Q4_K_M quant of a 1B to 4B model) from Hugging Face into a models folder.
  5. Chat in the terminal: ./build/bin/llama-cli -m models/your-model-Q4_K_M.gguf -t 4
  6. Or run a server with a web UI: ./build/bin/llama-server -m models/your-model-Q4_K_M.gguf -t 4 --host 0.0.0.0 --port 8080, then open http://<pi-ip>:8080 from any device.

Our llama.cpp guide explains all the flags, quantisation types and server options.

Best Models for Raspberry Pi

Pi RAM What to run Quant
4 GB 0.5B to 1.5B instruct models (Qwen, Llama 3.2 1B, Gemma small sizes) Q4_K_M
8 GB 3B to 4B instruct models as daily drivers; 7B to 8B if you can wait Q4_K_M
16 GB 4B fast, 7B to 8B for quality, small MoE models with few active parameters Q4_K_M or Q5_K_M

Tips for picking:

  • Choose instruct or chat versions, not base models.
  • Mixture-of-experts (MoE) models with small active parameter counts can run faster than dense models of the same total size, as long as the whole file fits in RAM.
  • Coding and maths need the biggest model you can tolerate; simple commands work fine with 1B.
  • Keep context short (2K to 4K tokens). Long context eats RAM and slows every reply.

Add a Web Chat UI

Typing in a terminal gets old. Two simple options:

  • llama-server’s built-in web UI: already included with llama.cpp. Start the server with --host 0.0.0.0 and open the Pi’s IP in a browser.
  • Open WebUI with Ollama: a full ChatGPT-style interface. It is heavier, so it suits 8 GB+ boards. Our Open WebUI + Ollama setup guide walks through it.

Make It Faster

  • Use Q4 quantisation. It roughly quarters memory use versus 16-bit weights with a small quality loss.
  • Use 4 threads (-t 4), one per core.
  • Keep it cool. Check throttling with vcgencmd get_throttled; anything other than 0x0 means power or heat trouble.
  • Boot from NVMe. Models load in seconds instead of minutes.
  • Run headless. Raspberry Pi OS Lite leaves more RAM for the model.
  • Close other apps and avoid running two models at once.
  • Add swap only as a safety net. A model that spills into swap becomes painfully slow and wears out SD cards.

The AI HAT+ 2 Option

Raspberry Pi’s AI HAT+ 2 launched in January 2026 for $130. It has a Hailo-10H accelerator rated at 40 TOPS (INT4) and its own 8 GB of RAM, connected to the Pi 5 over PCIe.

At launch it supported a small set of models, mostly at 1.5 billion parameters, including Qwen2.5-Instruct, Qwen2.5-Coder, Qwen2, Llama 3.2 and a DeepSeek R1 distill. Reviewers ran them through Docker with the hailo-ollama server. Raspberry Pi said larger models would follow in updates.

Pi 5 CPU only Pi 5 + AI HAT+ 2
Extra cost None $130
Model choice Any GGUF model that fits Only models compiled for Hailo
CPU load 100% while generating CPU stays free for other tasks
Also good for General use Camera and vision AI projects

If you want any model you like, stick with the CPU. If you are building a camera project or need the CPU free, the HAT is worth a look.

Project Ideas

  • Offline voice assistant: pair a small LLM with local speech-to-text. See our guide to running Whisper locally.
  • Home Assistant helper: point Home Assistant’s Ollama integration at the Pi’s API.
  • Private family chatbot on your home Wi-Fi with Open WebUI.
  • Kids’ homework helper that works with no internet.
  • Log summariser for your home server or network.

Curious about even smaller devices? See how to run an LLM on your phone.

Troubleshooting

Problem Likely cause Fix
“Killed” or out of memory Model too big for RAM Use a smaller model or lower quant; shorter context
Very slow, getting slower Thermal throttling Fit the Active Cooler; check vcgencmd measure_temp
Random reboots under load Weak power supply Use the official 27W 5V/5A supply
Model takes minutes to load Slow microSD Move to NVMe SSD or a fast A2 card
Install fails / “exec format error” 32-bit OS Reflash with 64-bit Raspberry Pi OS (uname -m = aarch64)
Can’t reach API from another PC Server bound to localhost Set OLLAMA_HOST=0.0.0.0 or --host 0.0.0.0
llama.cpp build fails Missing packages Install cmake, build-essential, libcurl dev package
Rambling or nonsense replies Base model or tiny model Use an instruct model; step up to 3B to 4B

My Take

A Raspberry Pi 5 with 8 GB of RAM, the Active Cooler and an SSD is a great little offline AI box. Stick to 1B to 4B models with Ollama for an easy start, and switch to llama.cpp when you want every bit of speed.

Do not expect desktop-GPU speed or GPT-level answers. With prices up sharply in 2026, check whether an old laptop or mini PC you already own would run models faster before buying a 16 GB Pi just for AI.

FAQ

Can a Raspberry Pi run ChatGPT?

Not ChatGPT itself, but it can run open models offline that work in a similar way. Small models (1B to 4B) run best.

Which Raspberry Pi is best for AI?

The Raspberry Pi 5 with 8 GB or 16 GB of RAM. The Pi 4 works for tiny models but is much slower.

Can a Raspberry Pi 5 run a 7B model?

Yes, on 8 GB or 16 GB boards at Q4, but replies are slow. A 3B to 4B model is a better balance.

Does Ollama work on Raspberry Pi?

Yes. Ollama supports 64-bit Arm Linux. Install it with the official script on a 64-bit OS.

Does the Pi GPU help with LLMs?

Not in practice with Ollama or standard llama.cpp builds; they run on the CPU. The AI HAT+ 2 adds a dedicated accelerator for supported models.

Do I need an internet connection?

Only to download the software and models. After that, everything runs offline.

How much power does it use?

Very little compared to a PC. The Pi 5 takes a 27W supply at most, and the whole setup can run all day on a few watts when idle.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Surprised woman holding a PI 5 card with the text Offline AI on a Pi 5, featured image for running an LLM on Raspberry Pi 5