AI INFRASTRUCTURE

I gave Qwen3.8's MTP drafter another 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That result sums up the whole experiment: the best local inference setup is rarely made from the individually "best" parts.

I wanted a dense 27B model, its full 262,144-token context, multimodal input, maximum useful quality, and speculative decoding on an NVIDIA RTX PRO 4000 Blackwell SFF with 24 GB of VRAM. The server also had to survive real agent work after printing model loaded . The experiment followed a hunch I had written about earlier : careful operation may matter as much as moving to a larger model.

The finished system averages 50.44 tok/s in the current ten-run production series. On a strict runtime A/B, the custom llama.cpp build reaches 55.40 tok/s versus 45.42 for clean master, a 21.97% gain. Against target-only greedy decoding, embedded MTP moves 21.19 to 59.46 tok/s, or 2.81 times the throughput. At the far end of a genuinely occupied 256K cache, it still produces 12.61 tok/s without an out-of-memory failure.

Those numbers came from different gates and should stay separate. Combining them into one heroic speedup would make a better headline and a worse benchmark.

The winning setup came from the fit between the quant, drafter, CUDA kernels, memory layout, and workload. No component won on its own.

The target was deliberately unreasonable

Qwen3.8 27B is a 64-layer dense model. Its repeating pattern contains three Gated DeltaNet layers followed by one full-attention layer, giving 48 recurrent layers and 16 conventional attention layers. It has a native 262,144-token context, a one-layer MTP head, and a separate 27-layer vision encoder.

The hardware is lopsided in a useful way:

• GPU0: RTX PRO 4000 Blackwell SFF , 24 GB GDDR7 with ECC, a 192-bit memory interface, 432 GB/s peak memory bandwidth, 24,467 MiB reported capacity, and sm120a. It holds the target, embedded MTP, recurrent state, graphs, and the 256K KV cache. • GPU1: RTX 2000 Ada, 15,996 MiB, sm89. It holds the F16 multimodal projector and other auxiliary services. • Runtime: Debian 13, CUDA 12.9.86, GCC 14.2, dual-architecture CUDA build.

Only the 16 full-attention layers grow a conventional KV cache with sequence length, which makes 256K less absurd than it first appears. With Q4 K and V, that cache costs roughly 4.25 GiB before allocator overhead. DeltaNet adds recurrent state and checkpoints instead. Four checkpoints were the useful minimum; the default 32 spent memory I needed elsewhere.

NVIDIA quotes 432 GB/s of peak bandwidth. That is a hardware ceiling rather than an application metric from llama.cpp, but it matters here. Autoregressive decode repeatedly streams quantized weights, and the 16 attention layers add increasingly expensive KV reads as context fills. This is why the same profile averages about 50 tok/s on the production task and 12.61 tok/s at the far end of a 261.5K-token cache.

Main content shown as an extended quote; the full article is at the source link.

Read original ↗ Content from Hacker News Frontpage（Community）