Skip to content
,

How to Chat With PDFs Locally and Offline: AnythingLLM and Open WebUI (2026)

9–13 minutes
Curious woman in a grey hoodie on a sunny park bench holding a white card reading PDF, hook text: Ask your PDF 100% offline

Want to ask questions about a contract, a research paper or a 300-page manual without uploading it to ChatGPT? You can do it fully offline on your own PC, with free tools and a local AI model.

This guide shows two easy ways to chat with PDFs locally: AnythingLLM Desktop (no technical setup) and Open WebUI with Ollama (more control). You will also learn how document chat really works, which settings make answers accurate, and how to fix the classic “the AI ignores my document” problem.

All steps and defaults come from the official AnythingLLM, Open WebUI and Ollama docs, checked in November 2026.

TL;DR

  • Easiest: install AnythingLLM Desktop, create a workspace, drag in your PDF, ask questions. It can run a local model with no extra setup.
  • More control: run Ollama plus Open WebUI, attach the PDF in chat, or build a Knowledge base and call it with #.
  • Small PDF: attach or pin it so the model reads the full text.
  • Big PDF or many files: embed them (RAG), so only the relevant chunks are sent.
  • Accurate answers only: use AnythingLLM’s Query mode, which answers only from your documents.
  • Ollama tip: its default context is 4,096 tokens. Raise it with OLLAMA_CONTEXT_LENGTH=8192 or more, or long documents get cut off.
  • Scanned PDFs need OCR first, or the tool sees blank pages.

How Local PDF Chat Works

There are two ways a local AI can “read” your PDF, and knowing the difference fixes most bad answers.

Method What happens Good for Weak spot
Full text in context The whole PDF text goes into the prompt Short PDFs, summaries, “read everything” questions Fails when the PDF is bigger than the model’s context window
RAG (embedding) The PDF is split into chunks, turned into vectors, and only the best-matching chunks are sent Long PDFs, many files, specific fact lookups Can miss details spread across the document

Open WebUI’s docs describe RAG simply: it splits documents into chunks, embeds them as vectors, stores them in a vector database, and at chat time retrieves only the relevant pieces for the model.

That is why “summarise the whole document” often works badly with RAG, while “what is the notice period in clause 7?” works well.

Everything here runs on your machine: the PDF parser, the embedding model, the vector database and the chat model. Nothing leaves your PC unless you choose a cloud model.

Which Tool Should You Use?

Feature AnythingLLM Desktop Open WebUI + Ollama
Setup One installer Install Ollama, then Open WebUI (pip, Docker or desktop app)
Local model engine Built in (llama.cpp based), or connect Ollama/LM Studio Ollama (or any OpenAI-compatible server)
Default embedder AnythingLLM Native Embedder all-MiniLM-L6-v2 (runs on CPU)
Answer only from docs Yes, Query mode Via system prompt or model settings
Full-text option Attach or pin documents Attach in chat
Reusable collections Workspaces Knowledge bases (Workspace > Knowledge)
Best for Beginners, quick private doc chat Power users, multi-user, ChatGPT-style UI

What Hardware You Need

AnythingLLM lists 16 GB RAM and an 8-core CPU as recommended, with a GPU of 8 to 12 GB VRAM or more for faster local models on Windows. Any Apple Silicon Mac works. Storage depends on the models you download.

Your PC Model size that runs well Experience
8 GB RAM, no GPU 1B to 3B (Q4) Usable for short PDFs, slow
16 GB RAM, 8 GB VRAM 7B to 9B (Q4) Good everyday doc chat
32 GB RAM, 12 to 16 GB VRAM 12B to 14B Better reasoning over long docs
24 GB+ VRAM or 32 GB+ Mac 27B to 32B Near cloud quality for most PDFs

Method 1: AnythingLLM Desktop

  1. Download AnythingLLM Desktop for Windows, Mac or Linux from anythingllm.com and install it.
  2. Pick an LLM on first launch. Choose the built-in AnythingLLM provider to download a local model inside the app, or choose Ollama or LM Studio if you already use them.
  3. Keep the default embedder (AnythingLLM Native Embedder) and default vector database (LanceDB). They are local and need no setup.
  4. Create a workspace, for example “Contracts” or “Thesis papers”. Each workspace has its own documents and chats.
  5. Add your PDF: drag it into the chat box, or click the + icon in the prompt input.
  6. If it is too big for the context window, AnythingLLM offers Cancel, Continue Anyway or Embed. Choose Embed for long files.
  7. Ask questions. Answers include source citations you can click to see which chunk was used.

Already running models in LM Studio? Our LM Studio beginner guide shows how to start its local server, which AnythingLLM can connect to.

AnythingLLM: Attach, Embed, Pin and Query Mode

These four features decide how good your answers are.

  • Attach: a dropped file is scoped to that workspace and that thread. By default its full text goes into the chat, so the model can use all of it. Hover over the + icon to see current context use.
  • Embed: the document is chunked and added to the workspace, so every thread in that workspace can use it. Best for large files and libraries.
  • Pin: pinning puts an embedded document’s full text into the context window. You can only pin documents that are already embedded, and pinned docs are left out of normal RAG results. Use it for short, critical files, since it costs more context and slows replies.
  • Chat modes: Chat mixes your documents with the model’s general knowledge. Query uses only your uploaded documents and tells you when it cannot find the answer. Agent mode can use tools and MCP servers.

For contracts, policies and anything where a made-up answer is risky, switch the workspace to Query mode.

Retrieval settings worth changing

Setting Default / advice When to change
Max Context Snippets Docs suggest about 4 to 6 for most models Raise a little for big-context models; too many adds noise
Document similarity threshold 20% minimum by default Set to No Restriction if answers say “not found” but the info is there
Search Preference (reranking) Off; adds about 100 to 500 ms Turn on for better chunk picks (LanceDB only)

Method 2: Open WebUI + Ollama

  1. Install Ollama from ollama.com and pull a chat model, for example ollama pull llama3.2. Our Ollama guide covers install and commands.
  2. Raise Ollama’s context length. Ollama’s docs say the default context window is 4,096 tokens. Start the server with OLLAMA_CONTEXT_LENGTH=8192 ollama serve (or higher if you have VRAM to spare) so documents are not cut off.
  3. Install Open WebUI. With Python 3.12: pip install open-webui then open-webui serve, and open http://localhost:8080. Or with Docker:
    docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

    then open http://localhost:3000. A desktop app is also available.

  4. Create the admin account on first launch. It is stored locally. (Full walkthrough: our Open WebUI + Ollama setup guide.)
  5. Pick your Ollama model from the model menu at the top of the chat.
  6. Attach the PDF by dropping it into the chat input, then ask your question. One-off attachments are processed for that chat only.

If answers are very slow, check that Ollama is actually on your GPU with ollama ps. Our Ollama not using GPU fix guide walks through every cause.

Open WebUI Knowledge Bases

  1. Go to Workspace > Knowledge and create a new collection, for example “Company policies”.
  2. Upload your PDFs into it. Open WebUI chunks and embeds them.
  3. Use it in any chat by typing # and picking the collection.
  4. Or bind it to a model in Workspace > Models, so that model always has access to those files.

Document settings (Admin Panel > Settings > Documents)

Setting Default Suggested start
Text Splitter character token
Chunk Size 1024 2000
Chunk Overlap 100 200
Top K 10 15 (or 5 for small local models)
Embedding model all-MiniLM-L6-v2 (CPU, about 500 MB RAM) Ollama embedder such as nomic-embed-text

After changing the embedding model, re-upload or re-index your documents. Vectors made by one embedding model cannot be searched with another.

Best Local Models for Document Chat

Need What to look for Why
Chat model A recent instruct model that fits in your VRAM at Q4 or better Follows “answer only from the context” instructions
Long PDFs A model with a large native context (32K+) Lets you raise context and pin more text
Embedding model embeddinggemma or nomic-embed-text in Ollama Small, fast, made for retrieval
Non-English PDFs A multilingual chat model and embedder English-only embedders match poorly

Ollama’s embeddings docs note that embedding vectors are typically 384 to 1,024 numbers long, depending on the model. You can test one with ollama run embeddinggemma "Hello world".

Tips for Accurate Answers

  • Ask specific questions. “What is the refund period in section 4?” beats “tell me about refunds”.
  • Use full text for summaries. Attach or pin short docs when you want an overview. RAG only sees a few chunks.
  • Tell the model the rules. Add to the system prompt: “Answer only from the provided documents. If the answer is not there, say so. Quote the page or section.”
  • Check the citations. Both tools show which chunks were used. If the cited chunk does not contain the answer, the answer is a guess.
  • Split huge PDFs by chapter. Smaller files give cleaner chunks and better matches.
  • Clean scanned PDFs. Run OCR first (many PDF editors and free OCR tools do this) so there is real text to read.
  • Keep tables simple. Complex tables often come out jumbled. Ask about one row at a time, or paste the table as text.

Troubleshooting

Problem Likely cause Fix
AI ignores my PDF Chunks filtered out or not attached to this chat Lower the similarity threshold; check the doc is embedded in this workspace or picked with #
“I can’t find that in the documents” but it is there Weak retrieval Rephrase with words from the PDF; raise snippets/Top K; turn on reranking
Answers invent facts Chat mode mixes in general knowledge Use Query mode (AnythingLLM) or a strict system prompt
Summary misses whole sections RAG only sends a few chunks Pin or attach full text; split the PDF
Blank or garbled text Scanned or image-only PDF OCR the PDF first
Output cut off or model forgets the doc Ollama 4,096-token default context Set OLLAMA_CONTEXT_LENGTH higher and restart Ollama
Very slow replies Model on CPU or too large Use a smaller model; check ollama ps for 100% GPU
Open WebUI can’t see Ollama models Docker cannot reach the host Keep --add-host=host.docker.internal:host-gateway, or set OLLAMA_BASE_URL
Old answers after swapping embedder Old vectors still stored Re-upload or re-embed the documents

My Take

If you just want to chat with a few PDFs privately, install AnythingLLM Desktop. It takes ten minutes, it works offline, and Query mode plus pinning gives you reliable answers.

If you already use Ollama or want a ChatGPT-style app for the whole family or team, Open WebUI is the better home. Raise Ollama’s context length, switch the text splitter to tokens and build Knowledge bases for the files you reuse.

Either way, remember the rule: full text for summaries, RAG for lookups, and always check the citation.

FAQ

Can I chat with a PDF offline?

Yes. AnythingLLM Desktop and Open WebUI with Ollama run the PDF parser, embedder and model on your own PC, with no internet needed after setup.

Is chatting with PDFs locally private?

Yes, as long as you pick a local model. Your files and chats stay on your machine. If you connect a cloud model, the relevant text is sent to that provider.

Which is easier, AnythingLLM or Open WebUI?

AnythingLLM Desktop is easier: one installer with a built-in model engine. Open WebUI needs Ollama and a Python, Docker or desktop install.

How big a PDF can I use?

With RAG, very large ones, even hundreds of pages, since only chunks are sent. For full-text reading, the PDF must fit in the model’s context window.

Why does the AI say the answer is not in my PDF?

Retrieval did not find a matching chunk. Use words from the document, lower the similarity threshold, or raise the number of snippets.

Can it read scanned PDFs?

Only after OCR. A scanned PDF is images, so convert it to searchable text first.

Do I need a GPU?

No, but it helps a lot. Small models run on CPU. For 7B+ models, 8 to 12 GB of VRAM or an Apple Silicon Mac gives smooth speed.

Can I chat with Word files and text files too?

Yes. Both tools accept common formats such as PDF, TXT and DOCX.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Curious woman in a grey hoodie on a sunny park bench holding a white card reading PDF, hook text: Ask your PDF 100% offline