Skip to content
,

AnythingLLM Guide (2026): Chat With Your Documents Privately, Agents and Setup

10–15 minutes
Smiling red-haired woman in a teal jacket holding an ANYTHINGLLM card in a bright kitchen, hook text: Chat with your files offline

AnythingLLM is a free, open-source app that turns your computer into a private ChatGPT that can read your own documents. You drop in PDFs, Word files or web pages, pick a local or cloud AI model, and ask questions.

Answers come back with citations to the exact source, and with a local model nothing leaves your PC.

It also does a lot more than document chat: AI agents that browse the web and save files, MCP tools, scheduled tasks, memories, a developer API and, in the Docker version, multiple users and an embeddable chat widget for your website.

This guide walks you through installing the desktop app or Docker version, setting up your first workspace, connecting Ollama or LM Studio, getting accurate answers from documents, using agents, and fixing common problems. Details come from the official AnythingLLM docs and GitHub repo (v1.20.0), checked in November 2026.

TL;DR

  • What: free, open-source “chat with your docs” app with agents, by Mintplex Labs. Current version 1.20.0.
  • Desktop: download from anythingllm.com/download for Windows, Mac or Linux. Single user, includes a built-in local LLM.
  • Docker: docker pull mintplexlabs/anythingllm, run on port 3001, open http://localhost:3001. Adds multi-user, passwords and a chat widget.
  • Recommended PC: 16 GB RAM and an 8-core CPU; a GPU with 8-12 GB VRAM on Windows for local models. Any M-series Mac works.
  • Agents: type @agent in a workspace, /exit to stop.
  • Privacy: turn off anonymous telemetry in the sidebar under Privacy, or set DISABLE_TELEMETRY=true.

What Is AnythingLLM?

AnythingLLM is an all-in-one AI application from Mintplex Labs. You connect any large language model, feed it your documents, and chat with them.

Under the hood it handles the hard parts of retrieval-augmented generation (RAG): splitting documents into chunks, turning them into embeddings, storing them in a vector database and pulling the right pieces into each answer.

Out of the box it uses its own native embedder and a built-in LanceDB vector database, so there is nothing extra to install. It reads PDF, TXT, DOCX and other common file types, and can also scrape websites.

Feature What it means for you
Workspaces Separate “rooms” with their own documents, settings and chat history
Citations Every answer shows which document chunks it used
Built-in LLM engine Run local models without installing Ollama (powered by Lemonade and llama.cpp)
Agents and MCP Web search, scraping, saving files, charts, SQL and any MCP server
Memories Automatic and user-managed memories about you or a workspace
Scheduled tasks Run prompts on a schedule with full agent abilities
Model routing Send chats to different providers based on rules you set
Developer API Use your workspaces from scripts and other apps

Desktop vs Docker

Desktop app Docker
Install One-click installer Docker command or Compose file
Users Single user, no login Multi-user with passwords and permissions
Built-in local LLM On by default Optional (ENABLE_BUILT_IN_LLM=true)
Access App window on your PC Any browser at port 3001
Website chat widget No Yes
White-labelling and themes for all users No Yes
Documents, agents, API, any LLM Yes Yes

Pick Desktop if it is just you on one computer. Pick Docker if you want a family or team server, browser access from other devices or a chatbot on your website.

System Requirements

Item Recommended (official)
RAM 16 GB
CPU Any 8-core CPU
Storage Varies with the size of the local models you download
GPU (Windows) Recommended for local models; 8-12 GB+ VRAM is great
Mac Any M-series chip handles local models; Intel Macs are slower
Docker server At least 2 GB RAM and 10 GB disk (when using a remote LLM)

If you only use cloud models such as OpenAI, Claude or Gemini, the requirements are much lower because the app itself is light. A PC without a GPU can also connect to a model running on another machine on your network.

Install the Desktop App

  1. Go to anythingllm.com/download and pick Windows, Mac or Linux.
  2. Run the installer. On Windows you may need to click More info then Run anyway if SmartScreen warns you.
  3. Open AnythingLLM. The onboarding screens ask you to choose an LLM. The built-in provider is the default and runs models locally.
  4. Pick a model to download (start with a small one, around 3-8B parameters, if you have 8 GB of VRAM or less).
  5. Keep the default embedder and vector database (AnythingLLM Native Embedder and LanceDB). They are private and need no setup.
  6. Create your first workspace when asked. You are ready to chat.

Note that the desktop app has no accounts or password. The docs warn against exposing it to the internet. Its “network discovery” option (off by default) only lets AnythingLLM Mobile, scripts or the API reach it on your local network.

Install With Docker

The Docker image supports both amd64 and arm64. The official guide mounts a folder on your PC so your data survives updates.

  1. Install Docker Desktop (Windows/Mac) or Docker Engine (Linux).
  2. Pull the image: docker pull mintplexlabs/anythingllm
  3. On Windows, open PowerShell and run the official command, which creates Documents\anythingllm and a .env file, then starts the container:
    $env:STORAGE_LOCATION="$HOME\Documents\anythingllm"; If(!(Test-Path $env:STORAGE_LOCATION)) {New-Item $env:STORAGE_LOCATION -ItemType Directory}; If(!(Test-Path "$env:STORAGE_LOCATION\.env")) {New-Item "$env:STORAGE_LOCATION\.env" -ItemType File}; docker run -d --rm -p 3001:3001 --cap-add SYS_ADMIN -v "$env:STORAGE_LOCATION`:/app/server/storage" -v "$env:STORAGE_LOCATION\.env:/app/server/.env" -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm
  4. On Linux or Mac, use the same idea with export STORAGE_LOCATION=$HOME/anythingllm, mkdir -p and touch for the .env file, then the same docker run flags.
  5. Open http://localhost:3001 and finish onboarding.
  6. For a shared server, turn on multi-user mode in settings and create accounts.

Important: inside Docker, localhost means the container, not your PC. To reach Ollama or LM Studio running on the host, use http://host.docker.internal:11434 (Ollama) or http://host.docker.internal:1234 (LM Studio). On Linux add --add-host=host.docker.internal:host-gateway to the run command.

To use the built-in engine in Docker, set ENABLE_BUILT_IN_LLM=true. On first boot it downloads the engine (a few MB) to engines/embedded-lemonade in your storage folder.

Choose Your AI Model

Change the model any time in Settings > AI Providers > LLM. You can also set a different model per workspace.

Option Best for Setup
Built-in (Lemonade + llama.cpp) Easiest local setup Pick and download a model in the app
Ollama If you already use Ollama Base URL http://127.0.0.1:11434, choose a pulled model
LM Studio Browsing GGUF models with a GUI Start LM Studio’s server, base URL http://127.0.0.1:1234/v1
KoboldCpp, LocalAI, text-generation-webui Other local back ends Point at their API URL
OpenAI, Anthropic, Gemini, xAI, DeepSeek, Mistral, Groq, OpenRouter and more Best quality, no GPU needed Paste your API key

New to local models? Our Ollama guide and LM Studio guide show how to download and run them, and KoboldCpp is another light option.

For embeddings, the native embedder is fine for most people. If you already run Ollama, nomic-embed-text is a popular alternative (it is the example used in the official Docker Compose file).

Create a Workspace and Add Documents

  1. Click New Workspace in the sidebar and give it a name, such as “Tax 2026” or “Product manuals”.
  2. Click the upload icon next to the workspace name.
  3. Drag in your files, or paste a URL to fetch a web page.
  4. Select the uploaded files and click Move to Workspace.
  5. Click Save and Embed. Wait while AnythingLLM chunks and embeds them.
  6. Ask a question in the chat. Expand the citations under the answer to see the source text.

You can also drag files straight into the chat box. Pin an important document (the pin icon in the document list) to send its full text with every message instead of only the best-matching chunks, which helps with short but critical files.

Get Better Answers From Your Documents

  • Use Query mode for strict answers. In the workspace’s chat settings, set the chat mode to Query so it only answers from your documents and says when it cannot find the answer. Chat mode mixes documents with the model’s general knowledge.
  • Keep workspaces focused. One topic per workspace gives the retriever less noise.
  • Raise the context snippets in the workspace’s vector database settings if answers miss details spread over many pages.
  • Adjust the similarity threshold if it returns “no relevant information” too often (lower it) or pulls in junk (raise it).
  • Use a bigger model. Small 1-3B models summarise poorly. 7-14B local models or a cloud model give much better answers.
  • Clean scanned PDFs. Image-only PDFs have no text to embed. Run OCR first.
  • Write a system prompt. Tell it who it is and how to answer, for example “Answer in bullet points and quote the page.”

For a side-by-side setup with Open WebUI as well, see our guide to chatting with PDFs locally.

Use AI Agents

Agents let the model use tools. Set them up once in the workspace’s Agent Configuration (choose an agent model and enable skills), then type @agent in the chat followed by your request. A popup shows which tools are on. Type /exit to end the agent session.

Default skill What it does
RAG Search Looks up what the workspace documents say and can save summaries to agent memory
Web Browsing Searches the internet for current information
Web Scraping Reads a site and embeds it into the workspace
Save Files Writes output to a file on your PC (you pick the location)
List and Summarize Documents Shows or summarises workspace files
Chart Generation Makes charts from a prompt or a CSV
SQL Agent Reads databases and runs SELECT queries (use a read-only user)

Example prompts:

  • @agent search the web for this week's RTX 5070 prices and save a summary to a file
  • @agent scrape https://example.com/docs and summarise the setup steps
  • @agent make a bar chart of monthly sales from sales.csv

The docs note that tool calling depends heavily on the model. Small 4-bit models often fail as agents, while the same model at 8-bit usually does better. If the agent ignores tools, try a higher quantisation, a larger model or a cloud model.

Beyond the built-in skills you can add MCP servers, build no-code Agent Flows and write custom agent skills.

Other Useful Features

  • Memories: automatic and manual memories about you or a workspace.
  • Scheduled tasks: run a prompt on a cron schedule, for example a morning news summary.
  • Dynamic model routing: send coding questions to one model and quick chats to a cheaper one.
  • Speech: built-in Whisper transcription for audio and video files, plus text-to-speech with the browser voice, local Piper, OpenAI or ElevenLabs.
  • Browser extension to send pages to a workspace, and an open-source Android app.
  • Developer API: create an API key in settings and query workspaces from your own code.

Privacy and Where Data Is Stored

OS Desktop storage folder
Windows C:\Users\<user>\AppData\Roaming\anythingllm-desktop\storage
Mac /Users/<user>/Library/Application Support/anythingllm-desktop/storage
Linux ~/.config/anythingllm-desktop/storage/

You can also open it from the app by clicking the version number at the top right. Back up this folder to keep your workspaces, documents and chats.

With a local LLM, the native embedder and LanceDB, your documents never leave your computer. If you pick a cloud model, the relevant chunks of your documents are sent to that provider with each question.

AnythingLLM collects anonymous usage telemetry by default. Turn it off in the sidebar under Privacy, or set DISABLE_TELEMETRY="true" in the server or Docker .env file.

Troubleshooting

Problem Likely cause Fix
Docker cannot connect to Ollama or LM Studio localhost points inside the container Use http://host.docker.internal:11434; on Linux add the host-gateway flag
Ollama models not listed Ollama not running or wrong URL Run ollama list, check the base URL http://127.0.0.1:11434
“No relevant information” answers Documents not embedded or threshold too high Click Save and Embed; lower the similarity threshold
Answers ignore my documents Chat mode mixes general knowledge Switch the workspace to Query mode, pin key files
PDF uploads but gives empty answers Scanned image PDF OCR the file first
Agent does not use tools Model too small or low quant Use an 8-bit or larger model, or a cloud model
Very slow replies Model too big for VRAM, running on CPU Pick a smaller model or a lower quant
Docker data lost after update No storage volume mounted Use the official command with STORAGE_LOCATION mounted
Browsing to the desktop port shows 404 Desktop has no web UI Use the app window, or Docker for browser access

AnythingLLM vs Other Local AI Apps

App Strength Pick it if
AnythingLLM Document chat, agents, multi-user server You mainly want to chat with files
LM Studio Model browser and fast local runtime You want to try many models
Jan Simple offline ChatGPT-style app You want a clean local chat app
Open WebUI Feature-rich web UI for Ollama You run Ollama on a server

See our Jan AI guide for the simplest offline chat option.

My Take

AnythingLLM is the easiest way I know to get private “chat with your documents” working. The desktop app with the built-in model takes minutes, citations make it trustworthy, and Query mode stops it inventing answers. For a home or office server, the Docker version with multi-user mode is excellent.

Start with one focused workspace and a 7-8B model. If answers are weak, try a larger local model or plug in a cloud API key for the hard questions.

FAQ

Is AnythingLLM free?

Yes. The desktop app and the Docker version are free and open source. Mintplex Labs also offers a paid hosted instance.

Does AnythingLLM work offline?

Yes. With the built-in LLM (or Ollama/LM Studio), the native embedder and LanceDB, everything runs on your computer.

Do I need Ollama for AnythingLLM?

No. The desktop app includes a built-in local LLM engine. Ollama is optional.

What file types can AnythingLLM read?

PDF, TXT, DOCX and many other document types, plus web pages, and audio or video through built-in transcription.

How much RAM does AnythingLLM need?

The docs recommend 16 GB RAM and an 8-core CPU. With only cloud models it needs much less.

How do I use agents in AnythingLLM?

Configure an agent model in the workspace, then type @agent and your request in the chat. Type /exit to stop.

Can several people use one AnythingLLM?

Yes, with the Docker version in multi-user mode. The desktop app is single user only.

Can I turn off AnythingLLM telemetry?

Yes. Disable it in the sidebar under Privacy, or set DISABLE_TELEMETRY to true in the .env file.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Smiling red-haired woman in a teal jacket holding an ANYTHINGLLM card in a bright kitchen, hook text: Chat with your files offline