Skip to content
,

DSpine for Qwen3.8-27B: 3x Faster Local Speeds With Tencent’s Draft Model (llama.cpp and SGLang Setup)

8–12 minutes
Woman at a home workstation holding a card reading DSPINE, hook 3x faster local Qwen

Qwen3.8-27B is one of the best models you can run on a single 24 GB card. Its weak spot is speed: with plain decoding it writes about 28 tokens per second on an RTX 3090.

DSpine, a small open “draft” model from Tencent, can push that to around 90 to 128 tokens per second on code and mixed chat, according to community tests.

The catch is that it is not a one-click install yet. This guide explains what DSpine is, which tools run it today (SGLang and a llama.cpp fork), the exact commands, the context and VRAM trade-off on 24 GB cards, and when it is not worth it.

TL;DR

  • What: a five-layer speculative decoding drafter for Qwen3.8-27B (tencent/Qwen3.8-27B-DSpine-b16), Apache 2.0. It guesses 16 tokens at once and the big model checks them in one pass.
  • Speed: RTX 3090 community test went from 28.2 tok/s to 90.1 (mixed), 128.7 (code) and 41.6 (prose). Tencent reports 3.84x throughput in its own agent replay at 8 concurrent requests.
  • Runs on: SGLang 0.5.22 (official, CUDA, BF16) or a community llama.cpp dspine branch with a 1.26 GB Q4_K_M GGUF. Not in upstream llama.cpp, Ollama, LM Studio or vLLM yet.
  • Cost: context. On 24 GB, max context drops from about 176K to 65K tokens with 15 drafted tokens.
  • Skip it if: you mostly write prose, use a Mac, or run AMD on Vulkan without tuning.

New to this model? Start with my guide to running Qwen3.8-27B on a 16 GB GPU, then come back here to make it faster.

What DSpine Is

DSpine is a speculative decoding drafter. A small, fast model proposes the next few tokens, and the big model (here Qwen3.8-27B) checks all of them in one forward pass. Every token the big model agrees with is kept for free.

If it disagrees, it keeps its own token and the draft restarts from there, so the final text comes from the big model.

What makes DSpine different from older drafters is that it drafts a whole block of 16 tokens in parallel, instead of one after another.

Per Tencent’s README, at every draft layer each position gets the predicted feature of the token before it, which keeps the guesses consistent with each other. The method is described in the paper “Draft in Parallel, Condition Through Depth” (arXiv 2609.36173).

Detail Value
Developer Tencent (Tencent/DSpine)
Target model Qwen3.8-27B (a separate drafter exists for Qwen3-8B)
Draft layers 5
Block size 16 tokens
Sliding window 8,192 tokens
Licence Apache 2.0
Released Paper and Qwen3-8B drafter in October 2026; Qwen3.8-27B support in November 2026

Important: the drafter is trained for one target. It works with Qwen3.8-27B and its quantised GGUF versions, but not with other models, and a fine-tune of Qwen3.8-27B would need a retrained drafter to get the same acceptance.

How Much Faster Is It?

The speed-up depends on how often the big model accepts the draft. Code and maths are predictable, so acceptance is high. Creative prose is not, so gains shrink.

Setup Without DSpine With DSpine Source
RTX 3090, Q4_K_M, code prompts 28.2 tok/s 128.7 tok/s (7.6 tokens accepted per step) Community test (llama.cpp branch)
RTX 3090, Q4_K_M, mixed prompts 28.2 tok/s 90.1 tok/s Community test
RTX 3090, Q4_K_M, prose 28.2 tok/s 41.6 tok/s (2.5 accepted per step) Community test
SGLang BF16, 8 requests, WorkBuddy Office replay 210 tok/s 806 tok/s (3.84x); DFlash2 665 Tencent
SGLang BF16, 8 requests, SWE-bench Pro replay 242 tok/s 713 tok/s; DFlash2 658 Tencent
DGX Spark, Q8_0 target – 29.5 tok/s vs DFlash2 29.6 llama.cpp PR test
RX 7800 XT (Vulkan), 15 drafted tokens 30.9 tok/s 19.1 tok/s (slower); 47.4 with 2 drafted Community test

Tencent’s own acceptance numbers for Qwen3.8-27B (tokens accepted per pass, DSpine vs DFlash2) are GSM8K 7.20 vs 5.69, MATH-500 6.49 vs 5.49, HumanEval 5.71 vs 4.87, MBPP 6.14 vs 5.04 and MT-Bench 4.48 vs 4.11. These are vendor figures.

One reviewer points out that DSpine drafts 16 tokens while DFlash2 drafts 8, which explains part of the gap.

The SGLang figures are total throughput across 8 users, not the speed one person sees. For a single chat on your own PC, the RTX 3090 numbers are the better guide.

Which Apps Support It

App DSpine for Qwen3.8-27B? Notes
SGLang 0.5.22 Yes (official) CUDA, BF16, FlashAttention 3. Installed by Tencent’s repo
llama.cpp (upstream) No Draft PR #31941 is open. Stock builds refuse the GGUF files
llama.cpp dspine branch Yes (community) Ajay9o9/llama.cpp, tested on CUDA
vLLM No No support reported as of 9 Nov
Ollama, LM Studio No Both build on llama.cpp, so wait for the upstream merge
Mac (MLX) Qwen3-8B only Tencent’s dspine_mlx covers full-attention Qwen3 targets, not the hybrid 27B

If you only use Ollama or LM Studio, bookmark this page. Once the llama.cpp PR merges, both apps usually pick it up in a later release. Our llama.cpp 1.0 guide explains how releases and tags work now.

Files and VRAM

You need two files: the normal Qwen3.8-27B GGUF (the target) and the DSpine drafter GGUF from aj9o9/Qwen3.8-27B-DSpine-b16-GGUF.

File Size Use
Qwen3.8-27B-Q4_K_M.gguf (target, e.g. Unsloth) about 16.5 to 17.1 GB The main model
Qwen3.8-27B-DSpine-b16-Q4_K_M.gguf 1.26 GB Recommended drafter
Qwen3.8-27B-DSpine-b16-Q8_0.gguf 2.20 GB Same acceptance, slower, more VRAM
Qwen3.8-27B-DSpine-b16-BF16.gguf 4.06 GB Only for making your own quants

Download them with the Hugging Face CLI:

hf download aj9o9/Qwen3.8-27B-DSpine-b16-GGUF Qwen3.8-27B-DSpine-b16-Q4_K_M.gguf --local-dir models

Qwen3.8-27B is a hybrid model with recurrent layers, so speculative decoding also keeps copies of that state. The community test measured about 157 MB per copy, or 2.35 GB for 15 drafted tokens. That is why context shrinks so much:

24 GB card, Q4_K_M target, q8_0 KV cache Max context
No drafter 176,128 tokens
DSpine Q4_K_M, 7 drafted tokens 98,304 tokens
DSpine Q4_K_M, 15 drafted tokens 65,536 tokens
DSpine Q8_0, 15 drafted tokens 40,960 tokens

On a 16 GB card the Q4_K_M pair (about 17.7 GB of weights) does not fit fully in VRAM, so DSpine is really a 24 GB-and-up trick for now.

Method 1: llama.cpp DSpine Branch (Single GPU)

This is the easiest way to try DSpine on a home PC with an NVIDIA card. You build a fork of llama.cpp that adds DSpine support. You need Git, CMake and the CUDA Toolkit installed.

  1. Clone and build the branch:
    git clone -b dspine https://github.com/Ajay9o9/llama.cpp.git
    cd llama.cpp
    cmake -B build -DGGML_CUDA=ON
    cmake --build build --config Release -j
  2. Start the server with both models:
    ./build/bin/llama-server \
      -m  Qwen3.8-27B-Q4_K_M.gguf \
      -md Qwen3.8-27B-DSpine-b16-Q4_K_M.gguf \
      --spec-type draft-dflash \
      --spec-draft-n-max 15 \
      -ngl 999 --spec-draft-ngl 999 \
      -fa on --jinja -c 32768 -np 1 \
      --cache-type-k q8_0 --cache-type-v q8_0 \
      --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

    On Windows the binary is build\bin\Release\llama-server.exe.

  3. Open http://127.0.0.1:8080 and chat, or point any OpenAI-compatible app at http://127.0.0.1:8080/v1.

What the key flags do:

  • -md: the draft model file.
  • --spec-type draft-dflash: the block-drafting mode that DSpine uses in this branch.
  • --spec-draft-n-max 15: how many tokens to draft. The default is 3, which wastes most of the 16-token block. 15 was best on NVIDIA in the branch’s tests.
  • -c 32768: set the context yourself. Auto-fit does not leave room for the drafter and the state copies.

This branch is a community project, not an official llama.cpp release. Keep your normal llama.cpp build for everyday use and run this one side by side.

Method 2: SGLang (Official)

Tencent’s official path is SGLang on Linux with CUDA and full BF16 weights. It suits a big GPU (the BF16 27B model alone is far more than 24 GB) or a server that handles several users.

  1. Install (Python 3.11+ and CUDA required; this installs SGLang 0.5.22):
    git clone https://github.com/tencent/dspine.git
    cd dspine
    python -m pip install -e .
  2. Serve (core flags from the README; the full command adds Mamba cache and CUDA graph options):
    sglang serve \
      --model-path /path/to/Qwen3.8-27B \
      --speculative-algorithm DSPINE \
      --speculative-draft-model-path tencent/Qwen3.8-27B-DSpine-b16 \
      --speculative-num-draft-tokens 16 \
      --tp-size 1 --dtype bfloat16 \
      --attention-backend fa3 --speculative-draft-attention-backend fa3 \
      --linear-attn-backend triton --enable-linear-replayssm-spec \
      --mamba-ssm-dtype float32 \
      --mem-fraction-static 0.80 --context-length 131072 \
      --max-running-requests 8 \
      --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
      --trust-remote-code

Copy the full command from the DSpine README before you run it, because it also sets the Mamba cache strategy, CUDA graph batch sizes and the sampling backend. The README describes this setup as a single GPU with up to eight concurrent requests.

Tuning: Draft Length and Context

  • NVIDIA (CUDA): start at --spec-draft-n-max 15.
  • Need more context or 2 slots on 24 GB: drop to 7. With -np 2, use 7.
  • AMD on Vulkan: try 2 or 3. At 15 the RX 7800 XT got slower than no drafter at all.
  • Mostly prose: try 3, or use the target’s own MTP head or DFlash2 instead.
  • Out of memory: lower -c first, then the draft length. Use the Q4_K_M drafter, not Q8_0.

A simple rule from the community test: each verify pass must return more than about 1.7 tokens to beat plain decoding on that GPU. If the server log shows an average acceptance under 2 at 15 drafted tokens, the drafter is costing you context without paying it back.

When DSpine Is Not Worth It

  • Story writing and long prose. Gains fall to around 1.5x and you lose a lot of context.
  • Very long documents. If you need 100K+ tokens of context on 24 GB, plain decoding keeps it.
  • Mac users. There is no published Metal result for the 27B drafter yet.
  • Ollama or LM Studio only. Wait for upstream support instead of building a fork.
  • Exact reproducibility. Even at temperature 0, output can differ slightly from runs without a drafter, mostly in prose and image descriptions, because batched checks can flip near-tied tokens.

For coding agents, tool calls and maths on a 24 GB NVIDIA card, it is one of the biggest free speed-ups available for this model right now.

Troubleshooting

Problem Fix
Stock llama.cpp refuses to load the drafter Expected. Build the dspine branch; upstream support is still a draft PR.
Log shows failed to allocate ... using host copies Context is too big. Lower -c, or drop --spec-draft-n-max to 7.
No speed-up at all Check that --spec-draft-n-max is set (default 3 is too low on CUDA) and that both models are on the GPU (-ngl 999 --spec-draft-ngl 999).
Slower than without DSpine (AMD/Vulkan) Lower draft length to 2 or 3.
Fast on code, slow on chat Normal: prose has low acceptance. Use a smaller draft length for chat.
Out of memory with two slots Use -np 2 with draft length 7, or one slot.
Draft used with a Qwen3.8-27B fine-tune gives little gain The drafter is trained on the base model. Fine-tunes need a retrained drafter.
SGLang install fails You need Python 3.11+, CUDA and an NVIDIA GPU supported by FlashAttention 3. The repo pins SGLang 0.5.22.

FAQ

Is DSpine free?

Yes. The code and drafter weights are Apache 2.0, and Qwen3.8-27B itself is Apache 2.0.

Does DSpine change the answers?

Mostly no, because the big model checks every token. Small differences can appear at temperature 0 where two tokens are almost tied, mainly in prose.

Can I use it in Ollama?

Not yet. Ollama uses llama.cpp, and DSpine is not in upstream llama.cpp. Watch PR #31941.

Will it work on a 16 GB GPU?

Not fully on the GPU. The Q4_K_M target plus drafter is about 17.7 GB of weights before any context.

What about Qwen3-8B?

Tencent also released tencent/Qwen3-8B-DSpine-b16, and that one has an official MLX path for Apple Silicon. Tencent reports 15.37 to 38.52 tok/s on an M4 Pro with Qwen3-8B in BF16.

Is this the same as DFlash2 or MTP?

They are all speculative decoding methods. DFlash2 is an earlier block drafter (8 tokens); MTP uses extra prediction heads built into some models. DSpine had higher acceptance than DFlash2 in Tencent’s tests.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Woman at a home workstation holding a card reading DSPINE, hook 3x faster local Qwen