Qwen3.8-27B 4-bit on a single DGX Spark (GB10 / SM121)

A measured serving recipe for 4-bit Qwen3.8-27B on one DGX Spark, with a side-by-side comparison of two 4-bit checkpoints and the FP8 baseline they are measured against. Every number in RESULTS.md came from the harness in bench/.

Headline: 75 tok/s single-stream (k=14), or 246 tok/s aggregate at 8-way concurrency (k=7, default batch budget). Those are two different configurations — draft depth trades one against the other, and no single setting wins both. See §3 of RESULTS.

This repository measures speed only. No quality evaluation was performed here; every accuracy figure below is cited from its publisher and unverified. Read LIMITATIONS.md before relying on any of it.

The drafting configuration is identical across the first three rows, so that difference is attributable to the weights alone. The last row adds draft depth, which requires --max-num-batched-tokens 16384.

Read this first: quantization only pays at low concurrency

Measured against FP8 on the same harness, DSpark k=7:

Single-stream decode is memory-bandwidth-bound, so fewer weight bytes is decisive. As the batch grows, one weight read serves many sequences, the workload turns compute-bound, and byte count stops mattering.

If you serve a fleet at c16, this entire repository buys you nothing — FP8 matches it and carries no quantization quality question. Quantize for interactive latency and low concurrency; do not quantize for fleet throughput.

4-bit also costs 7–9 % of prefill throughput at 32K–100K prompts (§4), because Marlin dequantization is compute-bound work that prefill pays for.

DFlash 2 does not run on either 4-bit checkpoint

DFlash 2 (Inco AI, August 2026) is the strongest drafter currently available for this model. Under vLLM it cannot be used with either checkpoint in this repo — it requires an unquantized target LM head, and both 4-bit builds fail the check (NVFP4 quantizes the head; AutoRound stores it in bf16 but vLLM still assigns it a quantized method). FP8 is the only build that works.

llama.cpp has no such restriction, and there it is the fastest single-stream configuration measured anywhere in this work:

But llama.cpp saturates at ~77 tok/s aggregate and its TTFT p50 reaches 4.0 s at c16, against vLLM's 220 tok/s and 1.1 s. The rule this repo already argues for quantization extends to the engine: llama.cpp for one interactive user, vLLM for anything else.

Separately — the drafter in most 4-bit recipes is the wrong one. On identical NVFP4 weights, free in-checkpoint MTP k=3 beats the Doopeworld DSpark k=7 drafter at every concurrency, by +15% at c1 rising to +49% at c16, while saving a 2.6 GiB drafter.

Full detail, the c1-c16 curves, the block-size ceiling and quality measurements: DFLASH2.md. References: SOURCES.md.

Read this next if you are choosing a quantization

GB10 has no native FP4 compute path. SM121 lacks the tensor-core support the FP4 GEMM kernels need, so 4-bit weights are dequantized through Marlin and multiplied in higher precision. Three consequences:

1. W4A4 buys nothing here. You pay the accuracy cost of quantized activations and receive no compute benefit, because the activation path is not executed in FP4. Prefer W4A16 — weight-only. The speedup on this device comes entirely from reading fewer bytes per token, not from faster math.
2. CUTLASS FP4 kernels are reported to emit silent garbage on this architecture. Not an error — wrong output. Keep them out of the path.
3. VLLM_MARLIN_USE_ATOMIC_ADD=1 is required. There is a documented race condition in the Marlin kernel on SM121 that produces incorrect output without it. Since 4-bit weights go through Marlin here, this sits directly in your critical path — and a kernel race looks exactly like "this quantization is bad quality".

Checkpoints compared

Two things worth noting from that table.

The MLP-only NVFP4A16 build is a quality upgrade, not a speed one. At 28.8 GiB it is the same size as FP8, so decode throughput is unchanged. Sparing attention, SSM, vision and lm_head is what keeps its divergence low — and what eliminates the size advantage.

The ModelOpt recipe has the strongest published evidence, and it is the only one of these reporting a code benchmark. If you need to justify a quantization to someone, HumanEval 39/40 is a better argument than an MMLU delta.

Which layers must not be quantized

Qwen3.8-27B is hybrid: 48 GatedDeltaNet (linear-attention) layers and 16 full-attention layers. Independent evaluation of the same-generation 27B found that NVFP4 builds which quantize the linear-attention path "consistently and significantly underperform", while builds keeping it in 16-bit show only subtle degradation.

The specific fix: linear attention can be quantized as long as in_proj_a and in_proj_b stay in higher precision. These are two tiny per-layer recurrence-control projections. Miss them and the model degrades in ways benchmarks may not surface but real work will.

If you quantize this model yourself, that is the single most important line in the recipe.

Quickstart

./serve.sh                      # NVFP4 + DSpark k=7 by default
MODEL=<int4-repo> ./serve.sh    # or the AutoRound int4 build
python bench/edit_bench.py
python bench/conc_bench.py

Where the speed actually comes from

Only a minority of the total speedup on this device is the quantization:

Six of the 9.5× is decode strategy on unchanged FP8 weights — speculative decoding is output-preserving by construction. Establish that baseline before reaching for quantization: it is free, reversible, and carries no accuracy question at all.

Quantization contributes the remaining 1.6×, and it is the only step here that changes what the model knows. Treat it accordingly — see Evaluation.

Evaluation

Published benchmark scores are necessary but not sufficient. A same-family checkpoint in this model line passed standard benchmarks and then failed a real code task badly enough to justify a blanket "16-bit weights only" policy for months. Benchmarks did not predict that; a concrete task did.

Before promoting any 4-bit build, run your own representative task — ideally a multi-file change or an API-precise piece of work, since that is where quantization damage shows up first, rather than one-shot generation which tends to look fine.

The speculative-decoding and prefix-caching configuration needs no such gate. It cannot change output.

License

MIT for the scripts and documentation in this repository. Model weights remain under their upstream licenses.