Skip to content
,

Open WebUI RAG Guide (2026): Chat With Your Documents and Knowledge Bases

6–9 minutes
Delighted silver-haired woman in a burgundy blazer in a modern office holding a card reading WEBUI, with the text Chat With Your Own Files

A local AI model knows a lot, but it doesn’t know your files: your notes, manuals, policies, invoices or research papers. Open WebUI fixes that with RAG (retrieval augmented generation). You upload documents, Open WebUI indexes them, and your model answers using what’s actually in them, with citations.

It works with Ollama or any OpenAI-compatible model, so you can chat with your documents completely offline.

This guide shows how to chat with files directly, build Knowledge bases in Workspace, attach knowledge to a custom model, tune the Documents settings (embedding model, chunk size, Top K, hybrid search, full-document mode), and fix the most common RAG problems. It follows the official Open WebUI docs as of November 2026.

TL;DR

  • Quick chat with a file: attach it with + or type # in the message box to pick files, knowledge or a URL.
  • Reusable library: Workspace > Knowledge > Create, then add files with Add Content.
  • Always-on knowledge: attach a Knowledge base to a custom model in Workspace > Models.
  • Biggest fix: raise the Ollama context length to at least 8192 (Settings > Models > Edit > Advanced Params). Ollama defaults to 4096 on GPUs under 24 GB.
  • Changed the embedding model? Click Reindex in Admin Settings > Documents.
  • Need the whole file? Use Using Entire Document or turn on full-text options.

How RAG Works in Open WebUI

  1. Extract: Open WebUI pulls the text out of your file (PDF, Word, Markdown and more).
  2. Chunk: it splits the text into small pieces.
  3. Embed: an embedding model turns each chunk into numbers that capture meaning, stored in a vector database.
  4. Retrieve: when you ask a question, the most relevant chunks are found.
  5. Answer: those chunks are added to your prompt, and the model answers with citations.

Small files can skip retrieval and be sent in full, which is often more accurate for short documents.

Before You Start

  • Open WebUI running and connected to Ollama. If not, follow our Open WebUI with Ollama setup guide.
  • A capable chat model in Ollama. See our Ollama guide.
  • Enough context length. This matters more than anything else (see Troubleshooting).

Method 1: Chat With a File

  1. Start a new chat and pick your model.
  2. Click + (More) > Attachments > Attach Files, or drag the file into the chat.
  3. Or type # in the message box to choose from Folders, Knowledge and Files, or paste a web or YouTube URL.
  4. Choose how the file is used in its options: Focused Retrieval (default, relevant chunks only) or Using Entire Document (the full text).
  5. Ask your question, e.g. “What is the notice period in this contract?”
  6. Check the citations under the answer to see which parts were used.

Tip: for a short file you want analysed completely, choose Using Entire Document. For long files, Focused Retrieval keeps the prompt small.

Method 2: Build a Knowledge Base

A Knowledge base is a reusable collection of documents, such as your company handbook or all your course notes.

  1. Go to Workspace > Knowledge and click Create.
  2. Give it a name and description, e.g. “HR Policies 2026”.
  3. Click Add Content (the + next to the search box) and choose Upload files, Upload directory, Sync directory, Add webpage or Add text content.
  4. Wait for processing to finish for each file.
  5. Use it in a chat by typing # and selecting the Knowledge base, or via + > Attachments > Attach Knowledge.

Sync directory is handy when your documents change often: point it at a folder and keep it updated.

Method 3: Attach Knowledge to a Custom Model

Want a “Policy Assistant” that always uses your handbook? Create a custom model with the knowledge built in.

  1. Go to Workspace > Models and create a new model (admins can also use Settings > Models > Edit).
  2. Pick a base model from Ollama.
  3. In the Knowledge field, search for and add your Knowledge base, notes or files.
  4. Add a system prompt such as “Answer only from the attached knowledge. If the answer isn’t there, say so.”
  5. Save and pick the new model in a chat.

Note: with native function calling (the default), attached knowledge isn’t automatically pasted into every prompt. The model calls knowledge tools to search it. If your model doesn’t use them well, keep the Knowledge Base tools enabled and add a hint in the system prompt.

Documents Settings Explained

Admins find these under Admin Settings > Documents.

Setting What it does Tip
Embedding engine / model Turns chunks into vectors. Default engine is built-in SentenceTransformers (all-MiniLM-L6-v2); Ollama, OpenAI and Azure OpenAI are also options A stronger embedding model improves retrieval. Reindex after changing it
Content extraction engine Pulls text from files. Default PDF reader is pypdf; Tika, Docling, Mistral OCR and others are available Use Tika or Docling for complex PDFs, tables or scans
Text splitter Character (default), token, or token_transformers Token is more predictable with LLM context limits
Chunk size / overlap Size of each piece and how much they overlap See the recipes below
Markdown header splitter Splits on headings Great for well-structured docs
Top K How many chunks are retrieved More gives more context but uses more tokens
Hybrid search Combines BM25 keyword search with vector search and reranking Helps with names, codes and exact terms
Top K Reranker How many chunks reach the model after reranking Used when hybrid search is on
Full Text Threshold Files under this size are sent whole Good for short documents
Always Use Full Text / Bypass Embedding and Retrieval Skips retrieval and sends full content Only with large-context models
RAG template The prompt that wraps retrieved text, using {{CONTEXT}} Don’t add {{QUERY}}; the question is added automatically

Recommended Settings by Setup

These starting points come from the Open WebUI RAG tutorial (community-written, hosted in the docs):

Setup Splitter Chunk size Overlap Top K
Local model, 8K context or less Token 1000 100 3 to 5
Mixed Token 1500 200 10
Cloud model, 32K+ context Token 2000 200 15 to 25

Changing chunk size or overlap doesn’t strictly require a reindex, but it’s recommended so existing documents use the new settings.

Chat With Web Pages

  1. Type # followed by the URL in the message box.
  2. Click the formatted URL that appears above the input to attach it.
  3. Ask your question about the page.

Pages full of menus and ads add noise, so a clean or “reader” version of the URL works better. Sites that need JavaScript may need the Playwright web loader (WEB_LOADER_ENGINE=playwright).

Troubleshooting

Problem Likely cause Fix
Model ignores the document or gives vague answers Ollama context too small (4096 on GPUs under 24 GB) Settings > Models > Edit > Advanced Params, set context to 8192 or more (16000+ for web pages). See our Ollama context length guide
Bad results after changing embedding model Old vectors don’t match Click Reindex in Admin Settings > Documents; re-upload files attached directly in chats
Model can’t “see” the content Extraction failed (scanned PDF, tables) Switch to Tika or Docling, or enable PDF image OCR
Only part of the document is used Retrieval picks a few chunks Use Using Entire Document, or Always Use Full Text
Custom model ignores its knowledge base Native tool calling not used well Keep Knowledge Base tools on and add a system-prompt hint
Error “‘NoneType’ object has no attribute ‘encode’” Embedding model setting broken Re-save the embedding model in Documents settings
Follow-up replies are slow Context re-sent each turn Set RAG_SYSTEM_CONTEXT=True
Upload crashes the server Multiple workers with default ChromaDB Set UVICORN_WORKERS=1
Exact names or codes not found Pure vector search Turn on hybrid search

Want a simpler desktop app for document chat? Compare with AnythingLLM or our chat with PDFs locally guide.

FAQ

Does Open WebUI RAG work offline?

Yes. With Ollama for chat and the built-in embedding model, everything runs on your machine.

What file types can I upload?

Common types include PDF, Word, Excel, PowerPoint, Markdown, text, CSV, JSON, HTML and code files. Admins can restrict extensions.

What’s the default embedding model?

The built-in SentenceTransformers engine with all-MiniLM-L6-v2. You can switch to an Ollama or OpenAI embedding model.

Why does my model say it can’t find the answer?

Usually the context length is too small or extraction failed. Raise the context length and check the extracted text.

Do I need to reindex after changing settings?

Yes after changing the embedding model. For chunk size and overlap it’s recommended.

Knowledge base or file attachment: which should I use?

Attach a file for one-off questions. Use a Knowledge base for documents you’ll ask about again and again.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Delighted silver-haired woman in a burgundy blazer in a modern office holding a card reading WEBUI, with the text Chat With Your Own Files