Nemotron 3.5 Transcribe: Run NVIDIA’s Open 40-Language Speech Model Locally (GGUF, VRAM, Setup)

NVIDIA’s Nemotron 3.5 Transcribe is an open-weights speech model that turns up to one hour of audio into text in 40 languages, with speaker labels, word timestamps and speech translation from one model.

It launched on 28 October 2026 aimed at H100-class servers. Since then, community GGUF builds and NVIDIA’s own NeMo-Speech.cpp runtime have made it usable on RTX cards, Macs and even 16 GB GPUs.

This guide covers the specs, benchmark numbers (all vendor-reported unless marked), which file to download for your GPU, step-by-step setup on Windows, Linux and Mac, prompts, how it compares with Whisper and Parakeet, and fixes for common problems.

What Is Nemotron 3.5 Transcribe?

Most speech models, like Whisper or Parakeet, are dedicated recognisers. Nemotron 3.5 Transcribe is a Speech-Augmented Language Model (SALM): an audio encoder feeds a full LLM, so you steer it with a text instruction as well as audio.

That is why one model can do several jobs that usually need a pipeline:

• Verbatim or clean transcripts with punctuation and capitalisation.
• Contextual transcription: pass names and jargon in the prompt so they are spelled right.
• Speaker attribution using a built-in Sortformer diarization encoder.
• Word timestamps via a CTC alignment head.
• Speech translation, language identification and audio question answering.
• PII tagging (a small part of training, so treat it as a helper, not a guarantee).

NVIDIA released it on 28 October 2026 on Hugging Face, NGC, build.nvidia.com and GitHub. A technical report was published the same day.

Specs at a Glance

Note: the model card itself is inconsistent in places: the name says 30B but the summary says 34B, and it says “40 languages” while its matrix lists 41 rows. We use the card’s detailed numbers.

Languages and Quality Tiers

NVIDIA sorts languages into three tiers. If you work in a “Supported” language, test on your own audio before relying on it.

• Speaker labels: listed for English, French, German, Hindi, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, Arabic and Chinese.
• Audio Q&A and spoken instructions: evaluated in English only.
• Weak spots on the card: Norwegian FLEURS WER 63.76%, Hindi-English code-switching 19.28%, Bengali-English 35.46%.

Benchmarks

All numbers below are vendor-reported by NVIDIA (word error rate, lower is better) unless marked otherwise.

Independent and Public Numbers

• Hugging Face Open ASR Leaderboard (2 Nov 2026, via Digital Applied): 6.10% average WER, rank 9. Whisper large-v3: 8.18% (rank 42). Parakeet TDT 0.6B v3: about 6.9-7.0%. Qwen3-ASR-1.7B: 6.14%.
• Artificial Analysis AA-WER: 2.8%, top among open-weight models at the time of AlphaSignal’s report.
• Speed on the board (RTFx): 472.62, against 835.62 for Qwen3-ASR-1.7B. Nemotron is accurate, not the fastest.

Warning: do not mix these scores. NVIDIA’s 4.71%, the board’s 6.10% and AA’s 2.8% use different test sets and scoring. NVIDIA’s headline speed (2,238 seconds of audio per second on one RTX PRO 6000) uses optimisations not in the open release; NVIDIA expects the open vLLM stack to be about 2x slower.

Which File to Download for Your GPU

The full BF16 model is about 67 GB and targets an H100 80 GB. For home use, grab the AtomicChat GGUF conversion, which splits the model into one shared encoder (1.67 GB) plus a decoder you pick by size.

Add the 1.67 GB encoder to every size above, plus working memory for context. A second GGUF set from prithivMLmods (Q4_K_M 25.4 GB, MXFP4_MOE 18.7 GB, plus a 1.47 GB mmproj) targets llama.cpp’s llama-mtmd-cli instead.

Tip: on a 16 GB card, the 19.68 GB NVFP4 file does not fit. Use AD-Q2_0_M with the reduced settings shown below. Not sure how much memory you have to spare? See our VRAM guide.

Run It Locally With NeMo-Speech.cpp

NeMo-Speech.cpp is NVIDIA’s own ggml-based C++ runtime for its speech models (Apache 2.0). It supports CUDA, Metal, Vulkan and CPU, and serves an OpenAI-compatible API.

What You Need

• GPU: see the table above. NVIDIA GPU for the batched CUDA path (CUDA 12.8+).
• Build tools (source builds): CMake 3.26+, Ninja, a C++17 compiler and SentencePiece 0.2.0+. Ubuntu 22.04’s packaged SentencePiece 0.1.96 fails to compile.
• Disk: about 15-37 GB depending on the decoder.
• Hugging Face CLI: pip install -U huggingface_hub gives you the hf command.

Step-by-Step Setup

1. Install the prebuilt runtime (easiest). Linux or macOS: curl -fsSL https://github.com/NVIDIA/NeMo-Speech.cpp/raw/main/scripts/install.sh | sh. Windows PowerShell: irm https://github.com/NVIDIA/NeMo-Speech.cpp/raw/main/scripts/install.ps1 | iex. Open a new terminal afterwards so PATH updates.
2. Or build from source with CUDA. Clone the repo, then run the three build commands in the code block below.
3. Download the encoder and one decoder into a models folder. Keep the encoder’s file name unchanged, because the decoder refers to it.
4. Transcribe a file with word timestamps, as shown below.
5. Start the server if you want an API that other apps can call.

git clone https://github.com/NVIDIA/NeMo-Speech.cpp
cd NeMo-Speech.cpp
git submodule update --init --depth 1 llama.cpp third_party/cpp-httplib
scripts/configure.sh cuda-server
cmake --build --preset cuda-server

# encoder + NVFP4 decoder (swap *AD-NVFP4.gguf for *AD-IQ4_NL.gguf or *AD-Q2_0_M.gguf)
hf download AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF --local-dir models --include "*Encoder-F16.gguf" --include "*AD-NVFP4.gguf"

# transcribe one file (JSON with word timestamps)
build/cuda-server/bin/nemo-speech transcribe audio.wav --model models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-AD-NVFP4.gguf --format json --word-times

# OpenAI-compatible server on port 8080
build/cuda-server/bin/nemo-speech serve --asr-model models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-AD-NVFP4.gguf --port 8080

If you used the prebuilt installer, call nemo-speech directly instead of build/cuda-server/bin/nemo-speech.

Useful Flags

• Speaker labels: add --diarize.
• Translation: add --translate-to de (any supported language code).
• Custom instruction: add --instruction "Transcribe with punctuation. Context terms: ComfyUI, Freakyjolly".
• Faster decoding: download the 1.15 GB DFlash2 file and pass --asr.salm.dflash_path with its path (batched CUDA path).

Settings for 16 GB GPUs

The uploader suggests these settings for AD-Q2_0_M on a 16 GB card. They were measured on an A100, so treat them as a starting point:

--asr.salm.max_sequences 2 --asr.salm.n_ctx_per_sequence 8192 --asr.salm.max_context 8192 --asr.salm.mtp_draft 0 --asr.salm.n_batch 512 --asr.salm.encoder_max_batch 2

Calling the Local Server

The server exposes /v1/audio/transcriptions and /v1/chat/completions. A quick test with speaker labels:

curl http://127.0.0.1:8080/v1/audio/transcriptions -F file=@audio.wav -F response_format=verbose_json -F diarization=true

Run the Full Model With vLLM

For servers with an H100, H200, A100 80 GB, L40S 48 GB (FP8) or RTX PRO 6000, NVIDIA’s reference path is vLLM 0.32.0 with the NeMo Speech plugin. It runs on Linux, or Windows through WSL2.

1. Clone NeMo Speech and create a Python 3.13 environment with uv, syncing the asr and vllm extras as shown on the model card.
2. Download the weights with hf download nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B.
3. Set the environment variables below, which allow hour-long audio.
4. Start the server with vllm serve, --max-model-len 65536, the MTP speculative config and the CTC timestamp middleware (full command on the model card).
5. Send audio as base64 through the OpenAI Python client.

export VLLM_PLUGINS=nemo_speechlm
export VLLM_MAX_AUDIO_DECODE_DURATION_S=3600
export VLLM_MAX_AUDIO_CLIP_FILESIZE_MB=180
export VLLM_USE_V2_MODEL_RUNNER=0

import base64
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
audio = base64.b64encode(open("audio.wav", "rb").read()).decode()
response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Transcribe the following audio."},
        {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio}"}},
    ]}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}, "skip_special_tokens": False},
)
print(response.choices[0].message.content)

Want to know how vLLM differs from llama.cpp-style runtimes in general? Our vLLM vs llama.cpp vs Ollama comparison explains the trade-offs.

Note: SGLang-Omni 0.2.0 can also serve /v1/audio/transcriptions without NeMo, but it has no word timestamps, diarization, translation or speculative decoding. Transformers support needs a source install of transformers.

Prompts for Transcripts, Speakers and Translation

NVIDIA recommends an empty system prompt for transcription and translation, with the instruction in the user message. Use temperature 0 and thinking off for these tasks.

Tip: a dense one-hour meeting can need about 20,000 output tokens. Raise or remove max_tokens, or the transcript stops halfway.

Nemotron 3.5 Transcribe vs Whisper vs Parakeet

Board figures as of 2 Nov 2026, compiled by Digital Applied. Here is who should use which:

• Nemotron 3.5 Transcribe: meetings, podcasts and interviews where you want speakers, timestamps and translation from one model, and you have 16 GB+ of VRAM.
• Parakeet TDT 0.6B v3: fast, cheap single-speaker transcription in European languages on small GPUs.
• Whisper large-v3: rare languages outside Nemotron’s 40, CPU-only machines, and the widest app support.
• Live captions: none of these is a true streaming model; NVIDIA’s separate Nemotron 3.5 ASR Streaming 0.6B fits that job better.

Building a full local audio toolkit? Pair this with our guides to local text-to-SFX models and ElevenLabs alternatives.

Limits, Licence and Privacy

• Offline only: internal 30-second chunks are an encoder detail, not a live mode. The NeMo-Speech.cpp --stream and --live options return errors for this model.
• One hour max: NVIDIA does not guarantee results beyond its 1-hour, 65K-token limits. Split longer files.
• Speaker tags are not identities: labels separate voices in one file; they do not tell you who someone is.
• Repetition loops: rare, but NVIDIA ships a regex guard that collapses repeated text. Check outputs on noisy audio.
• Phrase boosting: not supported in NeMo-Speech.cpp; put terms in the prompt instead.
• Licence: OpenMDW-1.1 allows commercial and non-commercial use. The repetition fix inside the repo is Apache 2.0 (see THIRD_PARTY_NOTICES.md).
• Training data: NVIDIA’s Granary v2 set, about 463,000 hours across 48 languages (per AiCybr’s summary of the report).

Privacy plus: running locally means recordings never leave your machine, which matters for calls, medical notes or client interviews. If you expose the server on a network, add authentication and file-size limits first.

Troubleshooting

My Take

This is the most useful open speech release of the autumn. Getting transcripts, speaker labels, timestamps and translation from one model removes the fragile Whisper-plus-pyannote pipelines many of us glue together.

The catch is size. At about 20 GB for a good 4-bit decoder, it wants a 24 GB card or a 32 GB RTX 5090. On 16 GB you can run it, but in 2-bit with tight settings, and the uploader’s 16 GB numbers were measured on a bigger GPU.

My advice: if you mostly transcribe clean English on a small GPU, Parakeet or Whisper are still fine. If you handle meetings, interviews or Indian and European languages with several speakers, Nemotron 3.5 Transcribe is worth the download. Check our best GPU for local AI guide if you are planning an upgrade.

FAQ

Is Nemotron 3.5 Transcribe free for commercial use?

Yes. It is released under OpenMDW-1.1, which the model card says allows commercial and non-commercial use.

Can I run Nemotron 3.5 Transcribe on a 16 GB GPU?

Yes, with the 13.57 GB AD-Q2_0_M GGUF plus the 1.67 GB encoder and reduced context settings. Expect a small accuracy loss, mainly in Chinese.

Does it work with Ollama or LM Studio?

Neither is listed in NVIDIA’s supported runtimes. Use NeMo-Speech.cpp, vLLM, SGLang-Omni or transformers instead.

Is it better than Whisper large-v3?

On the Open ASR Leaderboard it scores 6.10% average WER against 8.18% for Whisper large-v3. Whisper still covers more languages (99) and runs on far smaller hardware.

Can it transcribe live audio?

No. It is offline only and processes complete recordings up to one hour.

How many speakers can it label?

The prompt format uses up to four speaker tags (spk:0 to spk:3). Labels separate voices within one recording but do not identify people.

Does it support Hindi?

Yes. Hindi is in NVIDIA’s top “Leading” tier and supports speaker labels. Hindi-English code-switched speech is weaker, at 19.28% WER on MUCS.

Sources

• NVIDIA model card: nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B (Hugging Face)
• NVIDIA: Nemotron 3.5 Transcribe Technical Report (28 Oct 2026)
• NVIDIA NeMo-Speech.cpp GitHub repository and SALM docs
• AtomicChat: NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF (Hugging Face)
• prithivMLmods: NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF (Hugging Face)
• AlphaSignal: “AtomicChat makes NVIDIA’s Nemotron 3.5 Transcribe run on consumer GPUs” (28 Oct 2026)
• Digital Applied: “Nemotron 3.5 Transcribe vs Whisper and Parakeet Compared”
• AiCybr: “NVIDIA Nemotron 3.5 Transcribe: 40 Languages, GPU Requirements and Local Setup”