Skip to content
,

Run Any Hugging Face GGUF in Ollama (hf.co Guide)

8–11 minutes
Woman holding a card reading GGUF, hook any HF model in Ollama

The official Ollama library has a few hundred models. Hugging Face has tens of thousands of GGUF files: fine-tunes, new releases on day one, community quants in every size, and niche models for coding, roleplay, languages or tiny devices.

The good news is you do not need to download files by hand or write a Modelfile. Ollama can pull any public GGUF straight from Hugging Face with one command, and you can even pick the exact quant you want.

This guide covers the hf.co command, choosing a quant, renaming the model, running private repos, fixing chat templates, the manual Modelfile method for files you already have, and the errors people hit most.

TL;DR

  • Run any GGUF repo: ollama run hf.co/{user}/{repo} (also works with huggingface.co).
  • Pick a quant: add a tag, e.g. ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. Default is Q4_K_M when the repo has it.
  • Short name: ollama cp hf.co/... mymodel, then ollama run mymodel.
  • Private repos: add your Ollama public key (~/.ollama/id_ed25519.pub) to your Hugging Face SSH keys.
  • Already downloaded a .gguf? Make a Modelfile with FROM ./file.gguf and run ollama create.

New to Ollama? My Ollama guide covers install, models, commands and fixes in one place.

Why Run Hugging Face Models in Ollama

  • New models first. Community quantizers usually publish GGUF files within hours of a release, long before (or instead of) an official Ollama library entry.
  • Every size. Repos from quantizers like bartowski, unsloth and MaziyarPanahi offer many quants, from tiny 2-bit files to near-lossless Q8_0.
  • Fine-tunes. Coding, medical, multilingual and role-play fine-tunes are almost all on Hugging Face only.
  • Same Ollama workflow. Once pulled, the model works with ollama run, the API, Open WebUI and every app that already talks to Ollama.

The one requirement: the repo must contain GGUF files. A normal repo with only .safetensors weights will not work with this command. Look for “GGUF” in the repo name or in the file list.

Quick Start: One Command

Make sure Ollama is installed and running (my install guide covers Windows, Mac and Linux). Then:

ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

That is all. Ollama downloads the file, sets a chat template from the GGUF metadata, and opens the chat prompt. The format is always:

ollama run hf.co/{username}/{repository}

Use ollama pull instead of run if you only want to download it now and chat later. The full huggingface.co/... domain works too.

Find a GGUF Repo Worth Running

  1. Open the Hugging Face models page and filter by Libraries: GGUF, or search for the model name plus “GGUF”.
  2. Prefer repos from the model’s own team (for example an official “-GGUF” repo) or from well-known quantizers with many downloads.
  3. Read the model card: it tells you the base model, the licence, the recommended prompt format and often a table of quant sizes.
  4. Check the Files and versions tab to see the actual file sizes before you pull.

Hugging Face also shows a hardware hint on GGUF pages once you add your GPU in your account’s Local Apps and Hardware settings. My Hugging Face download guide explains repos, tokens and folders in more detail.

Choose a Quantization Tag

Without a tag, Ollama takes Q4_K_M if the repo has it, otherwise another sensible quant from the repo. To choose yourself, add the quant name after a colon:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:IQ3_M
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

The tag is not case-sensitive, so :q8_0 works too. You can also use the full file name as the tag:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-IQ3_M.gguf

Which quant to pick:

Quant Size vs. full model When to use it
Q8_0 About half of FP16 Plenty of VRAM, want near-original quality
Q6_K About 40% Very close to Q8, a bit smaller
Q5_K_M About 35% Good quality, good fit on mid-range cards
Q4_K_M About 30% The best default for most people
IQ3_M / Q3_K_M About 25% Squeezing a bigger model into small VRAM
IQ2 / Q2_K Under 20% Last resort, quality drops clearly

A simple rule: the file size plus 1-3 GB for context should fit in your VRAM. A bigger model at Q4 usually beats a smaller model at Q8. The full breakdown is in my GGUF quantization guide and my VRAM guide.

Use the “Use this model” Button

If you do not want to type repo names:

  1. Go to your Hugging Face account settings, open Local Apps, and enable Ollama.
  2. On any GGUF model page, click Use this model and choose Ollama. Copy the command it shows.
  3. For a specific quant, open the Files and versions tab, click a GGUF file to open the GGUF viewer, then use Use this model > Ollama from there. The command will include the quant tag.

Rename, List and Delete

Pulled models show up in ollama list with their full name, such as hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M. That is long to type, so copy it to a short name:

ollama cp hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M llama32-3b
ollama run llama32-3b

ollama cp does not copy the file again, it just adds a new name pointing to the same data. To remove a model and free disk space:

ollama rm hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

Files are stored in the normal Ollama models folder (OLLAMA_MODELS), not in your Hugging Face cache. More commands in my Ollama commands cheat sheet.

Chat Templates, System Prompts and Parameters

Ollama reads the tokenizer.chat_template stored inside the GGUF and picks the matching template from a list of common formats. For most modern models, that just works.

If replies look broken (the model talks to itself, prints tags like <|im_end|>, or never stops), the template is wrong or missing. You have two fixes:

Fix 1: Override it locally with a Modelfile

FROM hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M
PARAMETER temperature 0.6
PARAMETER num_ctx 16384
SYSTEM "You are a concise coding assistant."
ollama create llama32-coder -f Modelfile
ollama run llama32-coder

Add a TEMPLATE block if you need a custom prompt format. Ollama templates use Go template syntax, not Jinja. My Modelfile guide has ready-made examples.

Fix 2: Repo files (if you own the repo)

Repo owners can add three optional files that Ollama reads when pulling:

  • template: a Go template for the chat format.
  • system: a default system prompt.
  • params: sampling parameters in JSON, using the same names as Modelfile parameters.

Also check your context length: Ollama’s default can be as low as 4k tokens. See my Ollama context length guide.

Run Private or Gated Repos

For your own private repos (or ones in an organization you belong to), Ollama signs in with its SSH key:

  1. Show your Ollama public key:
    • Mac/Linux: cat ~/.ollama/id_ed25519.pub
    • Windows (PowerShell): Get-Content $env:USERPROFILE\.ollama\id_ed25519.pub
    • Linux service install: the key may be under /usr/share/ollama/.ollama/
  2. Copy the whole line.
  3. On Hugging Face, go to Settings > SSH and GPG Keys and click Add SSH key.
  4. Run ollama run hf.co/{username}/{private-repo} as usual.

For gated public models (where you must accept a licence), accept the terms on the model page first with the same account.

Manual Method: A Local .gguf File

Already have a GGUF on disk, or want to use one from LM Studio or llama.cpp? Point a Modelfile at it:

FROM ./Qwen3-8B-Q4_K_M.gguf
ollama create qwen3-8b-local -f Modelfile
ollama run qwen3-8b-local

Ollama copies the file into its own store, so you can delete the original afterwards if you need the space. This method also helps when a model needs a custom template from day one.

If you keep the same files for several apps, my LM Studio guide and llama-server guide show how they load GGUF files directly.

Use It from Apps and the API

An hf.co model is a normal Ollama model, so use its full name (or your short copy) anywhere:

curl http://localhost:11434/api/chat -d '{
  "model": "llama32-3b",
  "messages": [{"role": "user", "content": "Explain GGUF in one line"}]
}'

In Open WebUI, you can paste the hf.co/... name into the model pull box in the admin settings. Coding tools that use the OpenAI-compatible endpoint just need the model name. More in my Ollama API guide.

Troubleshooting

pull model manifest: file does not exist

  • Typo in the user or repo name. Copy it from the browser address bar.
  • The quant tag does not exist in that repo. Check the file list for the exact quant names.
  • The repo has no GGUF files, only safetensors. Find a “-GGUF” version of the model.
  • The repo is private or gated and your key or licence is not set up yet.

The repo only has split files (-00001-of-00003.gguf)

Very large models are often split into parts. If Ollama will not pull them, pick a smaller single-file quant, or download the parts and merge them with llama.cpp’s llama-gguf-split --merge tool, then use the manual Modelfile method.

Error: unknown model architecture

The model uses an architecture your Ollama version does not support yet. Update Ollama first. Brand-new architectures can take days or weeks to arrive, and llama.cpp often gets them first.

Model runs but answers are garbage

Usually a template problem (see the template section), or an extremely low quant such as Q2. Try Q4_K_M and a Modelfile with the template from the model card.

Download is very slow or stops

Run the same command again: Ollama resumes partial downloads. A wired connection and a quant that fits your VRAM keep both download and load times shorter.

Model runs on CPU and is slow

The quant is too big for your VRAM. Check ollama ps and follow my Ollama not using GPU fix.

FAQ

Do I need a Hugging Face account?

Not for public repos. You only need an account for private repos, gated models and the “Use this model” button.

Can I run any Hugging Face model in Ollama?

Any model with a GGUF version and an architecture Ollama supports. Models in other formats need converting to GGUF first.

Is hf.co the same as huggingface.co here?

Yes, both domains work in the command.

Where does Ollama store hf.co models?

In its normal models folder, the same place as library models. Change it with the OLLAMA_MODELS environment variable.

How do I update a model after the repo changes?

Run ollama pull with the same name again. Ollama only downloads layers that changed.

Is it safe?

GGUF files are data, not code, so they are much safer than old pickle files. Still prefer well-known quantizers and official repos, and read the licence before commercial use.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Woman holding a card reading GGUF, hook any HF model in Ollama