A local AI model knows a lot, but it doesn’t know your files: your notes, manuals, policies, invoices or research papers. Open WebUI fixes that with RAG (retrieval augmented generation). You upload documents, Open WebUI indexes them, and your model answers using what’s actually in them, with citations.
It works with Ollama or any OpenAI-compatible model, so you can chat with your documents completely offline.
This guide shows how to chat with files directly, build Knowledge bases in Workspace, attach knowledge to a custom model, tune the Documents settings (embedding model, chunk size, Top K, hybrid search, full-document mode), and fix the most common RAG problems. It follows the official Open WebUI docs as of November 2026.
- Quick chat with a file: attach it with + or type # in the message box to pick files, knowledge or a URL.
- Reusable library: Workspace > Knowledge > Create, then add files with Add Content.
- Always-on knowledge: attach a Knowledge base to a custom model in Workspace > Models.
- Biggest fix: raise the Ollama context length to at least 8192 (Settings > Models > Edit > Advanced Params). Ollama defaults to 4096 on GPUs under 24 GB.
- Changed the embedding model? Click Reindex in Admin Settings > Documents.
- Need the whole file? Use Using Entire Document or turn on full-text options.
How RAG Works in Open WebUI
- Extract: Open WebUI pulls the text out of your file (PDF, Word, Markdown and more).
- Chunk: it splits the text into small pieces.
- Embed: an embedding model turns each chunk into numbers that capture meaning, stored in a vector database.
- Retrieve: when you ask a question, the most relevant chunks are found.
- Answer: those chunks are added to your prompt, and the model answers with citations.
Small files can skip retrieval and be sent in full, which is often more accurate for short documents.
Before You Start
- Open WebUI running and connected to Ollama. If not, follow our Open WebUI with Ollama setup guide.
- A capable chat model in Ollama. See our Ollama guide.
- Enough context length. This matters more than anything else (see Troubleshooting).
Method 1: Chat With a File
- Start a new chat and pick your model.
- Click + (More) > Attachments > Attach Files, or drag the file into the chat.
- Or type # in the message box to choose from Folders, Knowledge and Files, or paste a web or YouTube URL.
- Choose how the file is used in its options: Focused Retrieval (default, relevant chunks only) or Using Entire Document (the full text).
- Ask your question, e.g. “What is the notice period in this contract?”
- Check the citations under the answer to see which parts were used.
Tip: for a short file you want analysed completely, choose Using Entire Document. For long files, Focused Retrieval keeps the prompt small.
Method 2: Build a Knowledge Base
A Knowledge base is a reusable collection of documents, such as your company handbook or all your course notes.
- Go to Workspace > Knowledge and click Create.
- Give it a name and description, e.g. “HR Policies 2026”.
- Click Add Content (the + next to the search box) and choose Upload files, Upload directory, Sync directory, Add webpage or Add text content.
- Wait for processing to finish for each file.
- Use it in a chat by typing # and selecting the Knowledge base, or via + > Attachments > Attach Knowledge.
Sync directory is handy when your documents change often: point it at a folder and keep it updated.
Method 3: Attach Knowledge to a Custom Model
Want a “Policy Assistant” that always uses your handbook? Create a custom model with the knowledge built in.
- Go to Workspace > Models and create a new model (admins can also use Settings > Models > Edit).
- Pick a base model from Ollama.
- In the Knowledge field, search for and add your Knowledge base, notes or files.
- Add a system prompt such as “Answer only from the attached knowledge. If the answer isn’t there, say so.”
- Save and pick the new model in a chat.
Note: with native function calling (the default), attached knowledge isn’t automatically pasted into every prompt. The model calls knowledge tools to search it. If your model doesn’t use them well, keep the Knowledge Base tools enabled and add a hint in the system prompt.
Documents Settings Explained
Admins find these under Admin Settings > Documents.
| Setting | What it does | Tip |
|---|---|---|
| Embedding engine / model | Turns chunks into vectors. Default engine is built-in SentenceTransformers (all-MiniLM-L6-v2); Ollama, OpenAI and Azure OpenAI are also options | A stronger embedding model improves retrieval. Reindex after changing it |
| Content extraction engine | Pulls text from files. Default PDF reader is pypdf; Tika, Docling, Mistral OCR and others are available | Use Tika or Docling for complex PDFs, tables or scans |
| Text splitter | Character (default), token, or token_transformers | Token is more predictable with LLM context limits |
| Chunk size / overlap | Size of each piece and how much they overlap | See the recipes below |
| Markdown header splitter | Splits on headings | Great for well-structured docs |
| Top K | How many chunks are retrieved | More gives more context but uses more tokens |
| Hybrid search | Combines BM25 keyword search with vector search and reranking | Helps with names, codes and exact terms |
| Top K Reranker | How many chunks reach the model after reranking | Used when hybrid search is on |
| Full Text Threshold | Files under this size are sent whole | Good for short documents |
| Always Use Full Text / Bypass Embedding and Retrieval | Skips retrieval and sends full content | Only with large-context models |
| RAG template | The prompt that wraps retrieved text, using {{CONTEXT}} | Don’t add {{QUERY}}; the question is added automatically |
Recommended Settings by Setup
These starting points come from the Open WebUI RAG tutorial (community-written, hosted in the docs):
| Setup | Splitter | Chunk size | Overlap | Top K |
|---|---|---|---|---|
| Local model, 8K context or less | Token | 1000 | 100 | 3 to 5 |
| Mixed | Token | 1500 | 200 | 10 |
| Cloud model, 32K+ context | Token | 2000 | 200 | 15 to 25 |
Changing chunk size or overlap doesn’t strictly require a reindex, but it’s recommended so existing documents use the new settings.
Chat With Web Pages
- Type # followed by the URL in the message box.
- Click the formatted URL that appears above the input to attach it.
- Ask your question about the page.
Pages full of menus and ads add noise, so a clean or “reader” version of the URL works better. Sites that need JavaScript may need the Playwright web loader (WEB_LOADER_ENGINE=playwright).
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| Model ignores the document or gives vague answers | Ollama context too small (4096 on GPUs under 24 GB) | Settings > Models > Edit > Advanced Params, set context to 8192 or more (16000+ for web pages). See our Ollama context length guide |
| Bad results after changing embedding model | Old vectors don’t match | Click Reindex in Admin Settings > Documents; re-upload files attached directly in chats |
| Model can’t “see” the content | Extraction failed (scanned PDF, tables) | Switch to Tika or Docling, or enable PDF image OCR |
| Only part of the document is used | Retrieval picks a few chunks | Use Using Entire Document, or Always Use Full Text |
| Custom model ignores its knowledge base | Native tool calling not used well | Keep Knowledge Base tools on and add a system-prompt hint |
| Error “‘NoneType’ object has no attribute ‘encode’” | Embedding model setting broken | Re-save the embedding model in Documents settings |
| Follow-up replies are slow | Context re-sent each turn | Set RAG_SYSTEM_CONTEXT=True |
| Upload crashes the server | Multiple workers with default ChromaDB | Set UVICORN_WORKERS=1 |
| Exact names or codes not found | Pure vector search | Turn on hybrid search |
Want a simpler desktop app for document chat? Compare with AnythingLLM or our chat with PDFs locally guide.
FAQ
Does Open WebUI RAG work offline?
Yes. With Ollama for chat and the built-in embedding model, everything runs on your machine.
What file types can I upload?
Common types include PDF, Word, Excel, PowerPoint, Markdown, text, CSV, JSON, HTML and code files. Admins can restrict extensions.
What’s the default embedding model?
The built-in SentenceTransformers engine with all-MiniLM-L6-v2. You can switch to an Ollama or OpenAI embedding model.
Why does my model say it can’t find the answer?
Usually the context length is too small or extraction failed. Raise the context length and check the extracted text.
Do I need to reindex after changing settings?
Yes after changing the embedding model. For chunk size and overlap it’s recommended.
Knowledge base or file attachment: which should I use?
Attach a file for one-off questions. Use a Knowledge base for documents you’ll ask about again and again.




