The official Ollama library has a few hundred models. Hugging Face has tens of thousands of GGUF files: fine-tunes, new releases on day one, community quants in every size, and niche models for coding, roleplay, languages or tiny devices.
The good news is you do not need to download files by hand or write a Modelfile. Ollama can pull any public GGUF straight from Hugging Face with one command, and you can even pick the exact quant you want.
This guide covers the hf.co command, choosing a quant, renaming the model, running private repos, fixing chat templates, the manual Modelfile method for files you already have, and the errors people hit most.
- Run any GGUF repo:
ollama run hf.co/{user}/{repo}(also works withhuggingface.co). - Pick a quant: add a tag, e.g.
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. Default isQ4_K_Mwhen the repo has it. - Short name:
ollama cp hf.co/... mymodel, thenollama run mymodel. - Private repos: add your Ollama public key (
~/.ollama/id_ed25519.pub) to your Hugging Face SSH keys. - Already downloaded a .gguf? Make a Modelfile with
FROM ./file.ggufand runollama create.
New to Ollama? My Ollama guide covers install, models, commands and fixes in one place.
Why Run Hugging Face Models in Ollama
- New models first. Community quantizers usually publish GGUF files within hours of a release, long before (or instead of) an official Ollama library entry.
- Every size. Repos from quantizers like bartowski, unsloth and MaziyarPanahi offer many quants, from tiny 2-bit files to near-lossless Q8_0.
- Fine-tunes. Coding, medical, multilingual and role-play fine-tunes are almost all on Hugging Face only.
- Same Ollama workflow. Once pulled, the model works with
ollama run, the API, Open WebUI and every app that already talks to Ollama.
The one requirement: the repo must contain GGUF files. A normal repo with only .safetensors weights will not work with this command. Look for “GGUF” in the repo name or in the file list.
Quick Start: One Command
Make sure Ollama is installed and running (my install guide covers Windows, Mac and Linux). Then:
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
That is all. Ollama downloads the file, sets a chat template from the GGUF metadata, and opens the chat prompt. The format is always:
ollama run hf.co/{username}/{repository}
Use ollama pull instead of run if you only want to download it now and chat later. The full huggingface.co/... domain works too.
Find a GGUF Repo Worth Running
- Open the Hugging Face models page and filter by Libraries: GGUF, or search for the model name plus “GGUF”.
- Prefer repos from the model’s own team (for example an official “-GGUF” repo) or from well-known quantizers with many downloads.
- Read the model card: it tells you the base model, the licence, the recommended prompt format and often a table of quant sizes.
- Check the Files and versions tab to see the actual file sizes before you pull.
Hugging Face also shows a hardware hint on GGUF pages once you add your GPU in your account’s Local Apps and Hardware settings. My Hugging Face download guide explains repos, tokens and folders in more detail.
Choose a Quantization Tag
Without a tag, Ollama takes Q4_K_M if the repo has it, otherwise another sensible quant from the repo. To choose yourself, add the quant name after a colon:
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:IQ3_M
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
The tag is not case-sensitive, so :q8_0 works too. You can also use the full file name as the tag:
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-IQ3_M.gguf
Which quant to pick:
| Quant | Size vs. full model | When to use it |
|---|---|---|
Q8_0 |
About half of FP16 | Plenty of VRAM, want near-original quality |
Q6_K |
About 40% | Very close to Q8, a bit smaller |
Q5_K_M |
About 35% | Good quality, good fit on mid-range cards |
Q4_K_M |
About 30% | The best default for most people |
IQ3_M / Q3_K_M |
About 25% | Squeezing a bigger model into small VRAM |
IQ2 / Q2_K |
Under 20% | Last resort, quality drops clearly |
A simple rule: the file size plus 1-3 GB for context should fit in your VRAM. A bigger model at Q4 usually beats a smaller model at Q8. The full breakdown is in my GGUF quantization guide and my VRAM guide.
Use the “Use this model” Button
If you do not want to type repo names:
- Go to your Hugging Face account settings, open Local Apps, and enable Ollama.
- On any GGUF model page, click Use this model and choose Ollama. Copy the command it shows.
- For a specific quant, open the Files and versions tab, click a GGUF file to open the GGUF viewer, then use Use this model > Ollama from there. The command will include the quant tag.
Rename, List and Delete
Pulled models show up in ollama list with their full name, such as hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M. That is long to type, so copy it to a short name:
ollama cp hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M llama32-3b
ollama run llama32-3b
ollama cp does not copy the file again, it just adds a new name pointing to the same data. To remove a model and free disk space:
ollama rm hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M
Files are stored in the normal Ollama models folder (OLLAMA_MODELS), not in your Hugging Face cache. More commands in my Ollama commands cheat sheet.
Chat Templates, System Prompts and Parameters
Ollama reads the tokenizer.chat_template stored inside the GGUF and picks the matching template from a list of common formats. For most modern models, that just works.
If replies look broken (the model talks to itself, prints tags like <|im_end|>, or never stops), the template is wrong or missing. You have two fixes:
Fix 1: Override it locally with a Modelfile
FROM hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M
PARAMETER temperature 0.6
PARAMETER num_ctx 16384
SYSTEM "You are a concise coding assistant."
ollama create llama32-coder -f Modelfile
ollama run llama32-coder
Add a TEMPLATE block if you need a custom prompt format. Ollama templates use Go template syntax, not Jinja. My Modelfile guide has ready-made examples.
Fix 2: Repo files (if you own the repo)
Repo owners can add three optional files that Ollama reads when pulling:
template: a Go template for the chat format.system: a default system prompt.params: sampling parameters in JSON, using the same names as Modelfile parameters.
Also check your context length: Ollama’s default can be as low as 4k tokens. See my Ollama context length guide.
Run Private or Gated Repos
For your own private repos (or ones in an organization you belong to), Ollama signs in with its SSH key:
- Show your Ollama public key:
- Mac/Linux:
cat ~/.ollama/id_ed25519.pub - Windows (PowerShell):
Get-Content $env:USERPROFILE\.ollama\id_ed25519.pub - Linux service install: the key may be under
/usr/share/ollama/.ollama/
- Mac/Linux:
- Copy the whole line.
- On Hugging Face, go to Settings > SSH and GPG Keys and click Add SSH key.
- Run
ollama run hf.co/{username}/{private-repo}as usual.
For gated public models (where you must accept a licence), accept the terms on the model page first with the same account.
Manual Method: A Local .gguf File
Already have a GGUF on disk, or want to use one from LM Studio or llama.cpp? Point a Modelfile at it:
FROM ./Qwen3-8B-Q4_K_M.gguf
ollama create qwen3-8b-local -f Modelfile
ollama run qwen3-8b-local
Ollama copies the file into its own store, so you can delete the original afterwards if you need the space. This method also helps when a model needs a custom template from day one.
If you keep the same files for several apps, my LM Studio guide and llama-server guide show how they load GGUF files directly.
Use It from Apps and the API
An hf.co model is a normal Ollama model, so use its full name (or your short copy) anywhere:
curl http://localhost:11434/api/chat -d '{
"model": "llama32-3b",
"messages": [{"role": "user", "content": "Explain GGUF in one line"}]
}'
In Open WebUI, you can paste the hf.co/... name into the model pull box in the admin settings. Coding tools that use the OpenAI-compatible endpoint just need the model name. More in my Ollama API guide.
Troubleshooting
pull model manifest: file does not exist
- Typo in the user or repo name. Copy it from the browser address bar.
- The quant tag does not exist in that repo. Check the file list for the exact quant names.
- The repo has no GGUF files, only safetensors. Find a “-GGUF” version of the model.
- The repo is private or gated and your key or licence is not set up yet.
The repo only has split files (-00001-of-00003.gguf)
Very large models are often split into parts. If Ollama will not pull them, pick a smaller single-file quant, or download the parts and merge them with llama.cpp’s llama-gguf-split --merge tool, then use the manual Modelfile method.
Error: unknown model architecture
The model uses an architecture your Ollama version does not support yet. Update Ollama first. Brand-new architectures can take days or weeks to arrive, and llama.cpp often gets them first.
Model runs but answers are garbage
Usually a template problem (see the template section), or an extremely low quant such as Q2. Try Q4_K_M and a Modelfile with the template from the model card.
Download is very slow or stops
Run the same command again: Ollama resumes partial downloads. A wired connection and a quant that fits your VRAM keep both download and load times shorter.
Model runs on CPU and is slow
The quant is too big for your VRAM. Check ollama ps and follow my Ollama not using GPU fix.
FAQ
Do I need a Hugging Face account?
Not for public repos. You only need an account for private repos, gated models and the “Use this model” button.
Can I run any Hugging Face model in Ollama?
Any model with a GGUF version and an architecture Ollama supports. Models in other formats need converting to GGUF first.
Is hf.co the same as huggingface.co here?
Yes, both domains work in the command.
Where does Ollama store hf.co models?
In its normal models folder, the same place as library models. Change it with the OLLAMA_MODELS environment variable.
How do I update a model after the repo changes?
Run ollama pull with the same name again. Ollama only downloads layers that changed.
Is it safe?
GGUF files are data, not code, so they are much safer than old pickle files. Still prefer well-known quantizers and official repos, and read the licence before commercial use.
Sources
- Hugging Face docs: Use Ollama with any GGUF model on the Hub (hf.co command, quant tags, default Q4_K_M, template/system/params files, private repos via SSH key)
- Ollama docs: Modelfile reference (FROM, PARAMETER, TEMPLATE, SYSTEM)
- Ollama FAQ (OLLAMA_MODELS location)




