How to Use LangChain with Ollama: Chat, RAG and Agents Locally (2026 Guide)

LangChain and Ollama are a perfect pair. Ollama runs the model on your own PC, and LangChain gives you the building blocks around it: prompts, memory, document search (RAG), tools and agents.

The connection between them is one small package, langchain-ollama. Once it is installed, every LangChain example you find online can run on a free local model instead of a paid API.

This guide shows you how to use LangChain with Ollama step by step on LangChain 1.x: setup, your first chat, prompts and chains, streaming, structured output, tool calling, a local agent, embeddings and a complete “chat with your documents” RAG app, plus fixes for the errors you will hit.

Why Use LangChain with Ollama

You can call Ollama directly with its REST API or Python library, as we showed in our Ollama API guide. That is the best choice for simple scripts.

LangChain earns its place when your app grows:

• Swap models easily. The same code runs with Ollama today and a cloud model tomorrow by changing one class.
• Ready-made parts. Document loaders, text splitters, vector stores and retrievers for RAG are all built in.
• Agents. create_agent handles the loop of calling tools, reading results and answering.
• Private and free. With Ollama, your documents and prompts never leave your machine, and there is no per-token bill.

Versions and What You Need

Hardware is whatever your model needs. A 4B to 8B model runs on most 8 GB GPUs or Apple Silicon Macs; bigger models need more memory. If you are new to Ollama, set it up first with our Ollama Docker guide or the normal installer.

Step 1: Install Everything

1. Install and start Ollama. Open http://localhost:11434 in a browser; it should say “Ollama is running”.
2. Pull a chat model: ollama pull gemma4 (or qwen3 for strong tool calling).
3. Pull an embedding model for RAG: ollama pull embeddinggemma (or nomic-embed-text).
4. Create a virtual environment: python -m venv .venv, then activate it (.venv\Scripts\activate on Windows, source .venv/bin/activate on Mac/Linux).
5. Install the packages:

pip install -U langchain langchain-ollama langchain-text-splitters pypdf

pypdf is only needed for the PDF example later. Check the install with pip show langchain-ollama.

Step 2: Your First ChatOllama Call

from langchain_ollama import ChatOllama

llm = ChatOllama(model="gemma4", temperature=0.3, validate_model_on_init=True)

reply = llm.invoke("Explain what an embedding is in two sentences.")
print(reply.content)

invoke returns an AIMessage. The text is in .content, and .response_metadata holds Ollama’s timing data such as eval_count and eval_duration.

validate_model_on_init=True checks that the model is actually pulled when you create the object, so a typo fails straight away instead of on the first call.

You can also pass a list of messages with roles:

messages = [
    ("system", "You are a helpful translator. Translate the user's sentence to French."),
    ("human", "I love programming."),
]
print(llm.invoke(messages).content)

Step 3: Prompts and Chains (LCEL)

LangChain lets you connect pieces with the | pipe operator. A prompt template, the model and an output parser make a reusable chain:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_ollama import ChatOllama

prompt = ChatPromptTemplate.from_messages([
    ("system", "You write short, friendly product descriptions."),
    ("human", "Write a 3-line description for: {product}"),
])
llm = ChatOllama(model="gemma4", temperature=0.7)
chain = prompt | llm | StrOutputParser()

print(chain.invoke({"product": "a USB-C desk lamp"}))

Chains also support .batch() to run several inputs and .stream() to stream the output.

Streaming and Chat History

Streaming makes local models feel much faster because words appear as they are generated:

for chunk in llm.stream("Give me 5 tips for a tidy desk."):
    print(chunk.content, end="", flush=True)

For a simple chatbot with memory, keep a list of messages and send it every time:

from langchain_core.messages import SystemMessage, HumanMessage

history = [SystemMessage("You are a concise tech helper.")]
while True:
    text = input("You: ")
    if text in ("quit", "exit"):
        break
    history.append(HumanMessage(text))
    answer = llm.invoke(history)
    history.append(answer)
    print("AI:", answer.content)

Long chats eventually overflow the model’s context window. Raise num_ctx (see settings) or trim old messages.

Structured Output with Pydantic

To get clean data instead of text, use with_structured_output. In langchain-ollama the default method is json_schema, which uses Ollama’s structured outputs to enforce the schema.

from pydantic import BaseModel, Field
from langchain_ollama import ChatOllama

class Recipe(BaseModel):
    name: str
    minutes: int = Field(description="Total cooking time")
    ingredients: list[str]

llm = ChatOllama(model="gemma4", temperature=0)
structured = llm.with_structured_output(Recipe)
r = structured.invoke("A quick vegetarian pasta for two.")
print(r.name, r.minutes, r.ingredients)

The other options are method="json_mode" (any JSON) and method="function_calling" (uses the tool-calling route). Keep temperature at 0 for the most reliable results.

Tool Calling

Tool calling lets the model ask your code to run a function. Decorate a function with @tool, bind it, and read tool_calls:

from langchain_core.tools import tool
from langchain_ollama import ChatOllama

@tool
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"It is 18°C and sunny in {city}."

llm = ChatOllama(model="qwen3", temperature=0).bind_tools([get_weather])
msg = llm.invoke("What's the weather in Lisbon?")
print(msg.tool_calls)  # [{'name': 'get_weather', 'args': {'city': 'Lisbon'}, ...}]

The model does not run the function; it only asks for it. To run tools automatically and feed the results back, use an agent.

Build a Local Agent with create_agent

LangChain 1.x replaced the old agent classes with one function, create_agent, which runs on LangGraph under the hood. It loops: call the model, run any tools it asks for, send results back, and stop when the model answers.

from langchain.agents import create_agent
from langchain_core.tools import tool
from langchain_ollama import ChatOllama

@tool
def add(a: float, b: float) -> float:
    """Add two numbers."""
    return a + b

@tool
def multiply(a: float, b: float) -> float:
    """Multiply two numbers."""
    return a * b

agent = create_agent(
    model=ChatOllama(model="qwen3", temperature=0),
    tools=[add, multiply],
    system_prompt="You are a careful maths assistant. Use the tools for every calculation.",
)

result = agent.invoke({"messages": [{"role": "user", "content": "What is (12.5 + 7.5) * 3?"}]})
print(result["messages"][-1].content)

Give each tool a clear docstring; that text is what the model reads to decide when to use it. Small models (under about 7B) often skip tools or pass wrong arguments, so start with a capable tool model.

Prefer building agents visually? Our n8n + Ollama agent guide does the same job without code.

Chat with Your Documents (Local RAG)

RAG (retrieval-augmented generation) finds the most relevant parts of your documents and gives them to the model as context. Here is a complete, fully local version using an in-memory vector store, so you do not need a database.

1. Load your text or PDF.
2. Split it into chunks of about 1,000 characters with some overlap.
3. Embed the chunks with OllamaEmbeddings and store them.
4. Retrieve the top chunks for each question.
5. Answer with ChatOllama, telling it to use only that context.

from pypdf import PdfReader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama, OllamaEmbeddings

# 1. Load
text = "\n".join(page.extract_text() or "" for page in PdfReader("manual.pdf").pages)

# 2. Split
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
chunks = splitter.split_text(text)

# 3. Embed and store
embeddings = OllamaEmbeddings(model="embeddinggemma")
store = InMemoryVectorStore.from_texts(chunks, embedding=embeddings)
retriever = store.as_retriever(search_kwargs={"k": 4})

# 4 + 5. Retrieve and answer
prompt = ChatPromptTemplate.from_template(
    "Answer using only this context. If the answer is not there, say you don't know.\n\n"
    "Context:\n{context}\n\nQuestion: {question}"
)
llm = ChatOllama(model="gemma4", temperature=0, num_ctx=8192)

def ask(question: str) -> str:
    docs = retriever.invoke(question)
    context = "\n\n---\n\n".join(d.page_content for d in docs)
    return llm.invoke(prompt.format_messages(context=context, question=question)).content

print(ask("How do I reset the device to factory settings?"))

Tips for better answers: use the same embedding model for indexing and searching, keep k between 3 and 6, and raise num_ctx so the retrieved chunks fit. If answers miss obvious facts, try smaller chunks (500 to 800 characters).

Important ChatOllama Settings

If you leave reasoning unset on a thinking model, its <think> text can show up inside the answer. Set reasoning=True to separate it or False to turn it off.

base_url also accepts http://user:password@host:11434 if your Ollama sits behind a proxy with basic auth.

Which Ollama Models to Use

Filter by “tools”, “thinking” and “embedding” on the Ollama model library to see which models support each feature. For hardware sizing, see our RTX 3060 12 GB for AI guide.

Troubleshooting

Out of disk space from too many models? Our guide to moving Ollama models to another drive fixes that.

My Take

Start with ChatOllama and a simple chain, then add RAG with the in-memory store before you reach for a database. That gets a private “chat with your PDFs” app running in under 50 lines.

For agents, model choice matters more than code. A solid tool-calling model like Qwen3 makes create_agent feel reliable; a tiny model makes it look broken. And because it is LangChain, you can always swap in a cloud model for the hard cases without rewriting your app.

FAQ

Is langchain-ollama free?

Yes. Both LangChain and Ollama are open source, and local models have no usage fees. You only pay for your own hardware and electricity.

What is the difference between ChatOllama and OllamaLLM?

ChatOllama works with chat messages and supports tools and structured output. OllamaLLM is a plain text-in, text-out model. Use ChatOllama for almost everything.

Do I need an API key for LangChain with Ollama?

No. The local Ollama server has no authentication. Only Ollama’s hosted cloud API needs a key.

Can I use LangChain with Ollama on another computer?

Yes. Set OLLAMA_HOST=0.0.0.0:11434 on the Ollama machine, restart it, and pass base_url="http://<ip>:11434" to ChatOllama and OllamaEmbeddings.

Which embedding model is best for local RAG?

embeddinggemma and nomic-embed-text are good, fast defaults. Whatever you choose, use the same model to build and search the index.

Does LangChain work with Ollama’s OpenAI-compatible endpoint?

Yes, via langchain-openai with base_url="http://localhost:11434/v1", but langchain-ollama is better because it supports Ollama-only settings like num_ctx, keep_alive and reasoning.

Why does my agent loop forever?

The model keeps calling tools without answering. Use a stronger tool model, make tool docstrings clearer, set temperature=0, and tell it in the system prompt to answer once it has the result.

Sources

• PyPI: langchain-ollama 1.1.0 and langchain 1.5.0
• langchain-ollama source (ChatOllama parameters, reasoning, base_url)
• LangChain docs: ChatOllama integration
• LangChain docs: create_agent
• Ollama docs: tool calling and embeddings