Skip to content
,

How to Use Vision Models in Ollama (2026): Images in CLI, API, Python and JS

6–9 minutes
Excited woman in glasses and a blazer next to a gaming PC holding a card with the Ollama llama logo, with the text Local AI Sees Your Images

Ollama isn’t just for text. With a vision model, your local AI can look at a photo, read text in a screenshot, describe a chart, check a receipt or explain an error message, all without sending the image to the cloud.

It takes one command to try. Point the model at an image file in the terminal, or send the image through the API from Python, JavaScript or any app that talks to Ollama.

This guide shows how to run a vision model in Ollama, pass images in the CLI, the REST API, Python and JavaScript, use them in a chat UI, pick the right model for your GPU, and fix the common problems. Commands follow the official Ollama docs as of November 2026.

TL;DR

  • Try it: ollama run gemma4 ./image.png what is in this image?
  • CLI: put the image path in the prompt; jpg and png work.
  • REST API: POST to http://localhost:11434/api/chat with an "images" array of base64 strings.
  • Python/JS libraries: pass a file path, raw bytes or base64 in images.
  • Chat UI: Open WebUI lets you drop images into the chat.
  • Raise context if long answers about big images get cut off.

What a Vision Model Can Do

A vision (multimodal) model accepts images as well as text. You give it a picture plus a question, and it answers about what it sees.

Task Example
Describe images “Describe this photo for alt text”
Read text (OCR) “What does the sign say?”
Explain screenshots “What does this error mean and how do I fix it?”
Charts and tables “Summarise the trend in this chart”
Documents “List the items and total on this receipt”
Sorting photos “Is this a cat, a dog or neither? Answer with one word”

Everything runs on your own PC, which is great for private photos and documents. New to Ollama? Start with our complete Ollama guide.

Which Vision Model to Use

The current Ollama docs use gemma4 as the example vision model. Ollama’s library has a Vision filter that lists every model that accepts images; check it for the latest options and sizes.

Model Command Notes
Gemma 4 ollama run gemma4 Used in the official vision docs
LLaVA 1.6 ollama run llava:7b / llava:13b / llava:34b Older classic; Ollama’s 2024 post notes up to 4x more pixels than v1.5 and better text reading
Other vision models See the Vision filter on ollama.com Pick a size that fits your VRAM

VRAM rule of thumb: pick the smallest model that does the job well. Images add extra tokens to the context, so leave headroom. If you run out, see our Ollama not using GPU fix.

Use Images in the Ollama CLI

  1. Open a terminal.
  2. Run the model with an image path in the prompt:
    ollama run gemma4 ./image.png what is in this image?
  3. Wait for the download the first time; ollama run pulls the model automatically.
  4. Read the answer, then ask follow-up questions in the same session.

You can also put the path inside a quoted prompt, as in Ollama’s original example:

ollama run llava "describe this image: ./art.jpg"

Use .jpg or .png files. On Windows, a full path such as C:/Users/you/Pictures/photo.jpg works; forward slashes avoid escaping problems.

Send Images Through the REST API

The REST API needs the image as a base64 string inside an images array on the message. This is the pattern from the official docs:

curl -L -o test.jpg "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg"
IMG=$(base64 < test.jpg | tr -d '\n')

curl -X POST http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4",
    "messages": [{
      "role": "user",
      "content": "What is in this image?",
      "images": ["'"$IMG"'"]
    }],
    "stream": false
  }'
  • "stream": false returns one JSON response instead of a stream.
  • The answer is in message.content.
  • The SDKs (below) accept file paths, URLs or bytes; only the raw REST API needs base64.

More API basics are in our Ollama API guide.

Python Example

Install the library with pip install ollama, then:

from ollama import chat

path = input('Path to an image: ')

response = chat(
    model='gemma4',
    messages=[{
        'role': 'user',
        'content': 'What is in this image? Be concise.',
        'images': [path],
    }],
)

print(response.message.content)

Instead of a path you can pass raw bytes (Path(path).read_bytes()) or a base64 string (base64.b64encode(Path(path).read_bytes()).decode()), which is handy when the image comes from a web upload.

Batch-describe a folder of photos

from pathlib import Path
from ollama import chat

for img in Path('photos').glob('*.jpg'):
    r = chat(model='gemma4', messages=[{
        'role': 'user',
        'content': 'Write one short alt-text sentence for this image.',
        'images': [str(img)],
    }])
    print(img.name, '-', r.message.content)

JavaScript Example

Install with npm install ollama, then:

import ollama from 'ollama'

const imagePath = '/absolute/path/to/image.jpg'
const response = await ollama.chat({
  model: 'gemma4',
  messages: [
    { role: 'user', content: 'What is in this image?', images: [imagePath] }
  ],
  stream: false,
})

console.log(response.message.content)

Use Vision in a Chat Interface

Prefer a ChatGPT-style window? Open WebUI connects to Ollama and lets you attach images to a message. Pick your vision model, click the attach button or drag the picture in, and ask your question. Our Open WebUI with Ollama setup guide walks through installation.

Want a model that always answers in a set style, such as “describe images for blind users”? Bake the instructions in with a Modelfile.

Practical Tasks and Prompts

Task Prompt
Alt text “Write a 15-word alt text for this image. No ‘image of’.”
Receipt “List each item with its price as a table, then the total.”
Screenshot error “Read the error in this screenshot and suggest a fix.”
Chart “What are the three main takeaways from this chart?”
Handwriting “Transcribe this handwritten note.”
Product photo “Write a 50-word product description from this photo.”
Classification “Answer only ‘indoor’ or ‘outdoor’.”

Working with PDFs rather than pictures? See how to chat with PDFs locally.

Tips for Better Results

  • Be specific. “Read the total on this receipt” beats “what is this?”
  • Crop to what matters. Small text in a huge photo is harder to read.
  • Use clear, well-lit images. Blurry photos mean guessy answers.
  • Ask for a format: table, JSON, one word, bullet list.
  • Double-check numbers. Vision models can misread digits; verify anything important.
  • Increase context length for long outputs. See our Ollama context length guide.

Local Vision Model vs OCR Tool vs Cloud AI

Ollama vision model Classic OCR tool Cloud AI (ChatGPT, Gemini)
Privacy Images stay on your PC Usually local Images are uploaded
Understands meaning Yes, can explain and reason No, only extracts text Yes, usually strongest
Cost Free after download Free or cheap Free tiers with limits, or paid
Speed Depends on your GPU Very fast Fast, needs internet
Best for Private photos, documents, batch jobs Pure text extraction from clean scans Hard images where accuracy matters most

Our take: use a local vision model for anything private or for large batches you’d rather not pay per image for. Use a dedicated OCR tool when you only need clean text from scans. Use a cloud model as a second opinion on difficult images.

Troubleshooting

Problem Likely cause Fix
Model ignores the image Text-only model Use a model with vision support (check the Vision filter)
“File not found” or no image detected Wrong path Use a full path; forward slashes on Windows
API returns an error about images Not base64, or line breaks in the string Base64-encode and strip newlines (tr -d '\n')
Very slow Model on CPU or too big Use a smaller model; check GPU use with ollama ps
Out of memory Model plus image too large for VRAM Smaller model or quantization
Answer cut off Context too small Raise num_ctx
Wrong text read Low resolution or blur Crop and use a sharper image
WebP or HEIC not working Format not supported Convert to jpg or png

FAQ

Can Ollama read images?

Yes, with a vision model such as gemma4 or LLaVA. Pass the image path in the CLI or send it through the API.

What image formats does Ollama support?

Ollama’s docs mention .jpg and .png. Convert other formats first.

Does it work offline?

Yes. After the model downloads, everything runs locally and your images never leave your PC.

How do I send an image with the API?

Add an images array with a base64 string to your message when you POST to /api/chat.

Can I send several images at once?

The images field is an array, but Ollama’s docs only show one image. Support for several depends on the model, so test with yours.

Which vision model is best?

It depends on your GPU and task. Start with the model used in Ollama’s docs, then compare others from the Vision filter on your own images.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Excited woman in glasses and a blazer next to a gaming PC holding a card with the Ollama llama logo, with the text Local AI Sees Your Images