This post was made with AI. I tried to remove as much slop as possible and keep it straight to the point to save your time as I know how annoying AI slop posts can be, but I still wanted to retain all the details so it can be used as a resource for comparison with other future quants, I advise that any humans just skim through it or read the 1st section and run the balanced profile with a context smart harness like pi or deepseek harness (apparently really good for this) Optimizing Qwen3.6 / Qwen3.8 27B on 16GB VRAM: Complete Experimental Log, Benchmarks, and Deployment Guide This document records the complete set of benchmarks, quantization evaluations, KV cache sweeps, speculative decoding experiments, and context scaling tests conducted across multiple GPU architectures to determine the optimal configuration for running Qwen 27B hybrid models within a 16GB VRAM constraint. 1. Primary Recommendations (Quick Reference) 1.1 Balanced Profile (Recommended Default)

• Model: Qwen3.8-27B-IQ4_XS-pure-MTP.gguf (14.56 GB)

• Inference Engine: beellama.cpp

• KV Cache Format: kvarn4 (4-bit variance-normalized KVarN)

• Speculative Decoding: Native MTP at draft depth 2 ( --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.0 )

• Maximum Context (100% VRAM): 32,768 to 48,000 tokens

• Quality Metric: 92.55% Top-1 token match, 0.1169 mean KL divergence vs. uncompressed Q8_0 reference

1.2 Extended Context Profile (For Context > 48K)

• Model: Qwen3.8-27B-AD-IQ3_S-IQ3_XXS.gguf (12.98 GB)

• KV Cache Format: kvarn4

• Speculative Decoding: Native MTP at draft depth 2

• Maximum Context (100% VRAM): ~72,000 tokens (+50% context headroom)

• Quality Metric: 89.85% Top-1 token match, 0.2282 mean KL divergence vs. uncompressed Q8_0 reference

1.3 Recommended Server Launch Command export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH ./build/bin/llama-server \ -m ./models/Qwen3.8-27B-IQ4_XS-pure-MTP.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.0 \ --cache-type-k kvarn4 \ --cache-type-v kvarn4 \ -c 32768 \ -ngl 999 \ -fa on \ -np 1 \ --fit off \ -t 8 \ --host 127.0.0.1 \ --port 8080 \ --api-key your-key-here 2. Quantization Quality Benchmarks 2.1 Final Evaluation vs. Q8_0 Gold Reference Evaluation conducted using llama-perplexity against a gold reference generated from uncompressed Qwen3.8-27B-Q8_0.gguf (29.05 GB, PPL 3.9616).

• Corpus: 20 randomly sampled reasoning and math problems from canada-quant/hy3-w4a16-mtp-calibration (51,831 bytes, 24 chunks of 512 tokens).

• KV Cache: kvarn3 throughout to maintain exact numerical consistency.

• Logits Reference: Published on HuggingFace Hub as MaxDevv/Qwen3.8-27b-Q8-Logits-for-kld (3.04 GB).

Model Quantization File Size PPL ( hy3 , 512 ctx) Mean KLD vs. Q8 (Lower = Better) Top-1 Match vs. Q8 (Higher = Better) RMS Δp Notes Q8_0 (Gold Baseline) 29.05 GB 3.9616 ± 0.131 0.0000 100.0% — Uncompressed reference baseline Qwen3.8 IQ4_XS-pure 14.56 GB 4.1474 ± 0.142 0.1169 ± 0.011 92.55 ± 0.34% 8.87% Highest distribution fidelity; recommended Atomic AD-IQ3_S-IQ3_XXS 12.98 GB 3.9594 ± 0.129* 0.2282 ± 0.014 89.85 ± 0.39% 12.87% Best sub-4-bit quant; recommended for high context Atomic AD-IQ3_XXS 12.08 GB 4.8501 ± 0.183 0.3100 ± 0.018 87.93 ± 0.42% 13.91% Plain 3-bit dynamic; moderate degradation Ridge 3.7bpw 12.60 GB 4.5645 ± 0.163 0.3554 ± 0.019 86.44 ± 0.44% 15.16% Fixed 3.7bpw; notable distribution divergence Qwen3.6 IQ3_XXS-MTP 12.20 GB 4.6392 ± 0.185 0.9034 ± 0.035 78.79 ± 0.52% 22.99% Evaluated on Qwen3.6 weights; severe quality drop \ Note on Atomic PPL: The* hy3 dataset serves as the calibration source for the Atomic/w4a16 quantization pipeline. This artificially flatters its raw PPL score on this corpus. The KLD (0.2282) and Top-1 match (89.85%) reflect its true, unbiased generalization accuracy. 2.2 Preliminary Wikitext-2 Quality Benchmarks Earlier exploratory testing evaluated quants against a 4-bit IQ4_XS-pure local reference over 20 chunks of Wikitext-2 (512 context): Model Quantization File Size Perplexity (Wikitext-2) Mean KLD vs. 4-bit Ref Evaluation Qwen3.8 IQ4_XS-pure 14.56 GB 7.1252 ± 0.249 0.0000 (Ref) Baseline 4-bit reference Atomic AD-IQ3_S-IQ3_XXS 12.98 GB 7.3215 ± 0.256 0.0567 Retains high linguistic consistency Ridge 3.7bpw 12.60 GB 7.6710 ± 0.267 0.1373 +7.7% PPL degradation vs. 4-bit Qwen3.6 IQ3_XXS-MTP 12.20 GB 7.8892 ± 0.296 0.3552 Demonstrates clear sub-3.5bpw degradation 2.3 External Baseline Reference: Comprehensive Evaluation of Qwen3.6-27B Quantizations To contextualize experimental results against baselines, the table below compiles the empirical benchmark dataset published by Quesma (Piotr Migdał, Do Qwen3.6 27B quantizations break the pelican? , July 2026). This dataset tracks statistical distribution drift, downstream mathematical reasoning, terminal agent execution, generative SVG visual fidelity, and M5 Max execution throughput across 24 distinct quantization formats derived from Qwen3.6-27B-BF16 . Master Quantization Reference Table Quantization Format Family Size (GB) Mean KLD vs. BF16 Top-1 Match (Temp 0) Token Collision (Temp 1) Wikitext-2 PPL AIME-120 Solved (%) Terminal-Bench 2.1 Passed (%) Pelican Bradley-Terry Score Generation Speed (M5 Max) BF16 (Reference) Uncompressed 54.7 0.0000 100.0% 48.48% 6.40 70.8% (85/120) 38.2% (34/89) +1.76 ± 9.05 24.9 tok/s UD-Q8_K_XL Unsloth Dynamic 35.8 0.00038 99.4% 48.47% 6.39 70.0% ($p=1.00$) — +1.27 ± 3.53 31.8 tok/s FP8 vLLM Float 30.9 0.01700 96.7% — — — — -0.21 ± 1.90 — Q8_0 Standard K 29.0 0.00049 99.3% 48.46% 6.39 67.5% ($p=0.45$) 36.0% ($p=0.75$) +1.19 ± 2.09 30.8 tok/s UD-Q6_K_XL Unsloth Dynamic 26.0 0.00140 98.8% 48.44% 6.40 65.8% ($p=0.18$) — +1.70 ± 1.07 33.1 tok/s NVFP4 (Unsloth) vLLM Float 23.3 0.04400 92.2% — — — — +0.27 ± 0.81 — Q6_K Standard K 22.9 0.00210 98.3% 48.46% 6.41 64.2% ($p=0.12$) 30.3% ($p=0.17$) +2.54 ± 2.59 33.0 tok/s NVFP4 (NVIDIA) vLLM Float 21.9 0.03900 92.1% — — — — +1.77 ± 2.63 — UD-Q5_K_XL Unsloth Dynamic 20.4 0.00460 97.4% 48.41% 6.40 70.8% ($p=1.00$) — +1.39 ± 1.02 30.3 tok/s Q5_K_M Standard K 19.8 0.00520 97.2% 48.38% 6.40 65.0% ($p=0.12$) 29.2% ($p=0.13$) +1.12 ± 0.88 29.5 tok/s Q5_K_S Standard K 19.3 0.00580 97.1% 48.39% 6.40 67.5% ($p=0.52$) — +1.59 ± 0.80 30.4 tok/s UD-Q4_K_XL Unsloth Dynamic 17.9 0.01300 95.8% 48.29% 6.45 70.8% ($p=1.00$) — +0.48 ± 0.85 32.8 tok/s Q4_1 Legacy Q4 17.5 0.01800 94.8% 48.19% 6.43 64.2% ($p=0.14$) — +0.82 ± 4.09 34.4 tok/s Q4_K_M Standard K 17.1 0.01700 95.1% 48.29% 6.45 65.8% ($p=0.21$) 40.4% ($p=0.79$) +1.09 ± 0.90 32.7 tok/s IQ4_NL Importance Matrix 16.3 0.01800 94.8% 48.19% 6.45 66.7% ($p=0.33$) — +2.54 ± 6.36 32.7 tok/s Q4_0 Legacy Q4 16.1 0.03500 93.0% 48.10% 6.55 64.2% ($p=0.15$) — +1.31 ± 0.65 34.7 tok/s Q4_K_S Standard K 16.1 0.01900 94.7% 48.23% 6.46 63.3% ($p=0.06$) — +2.18 ± 0.98 33.5 tok/s IQ4_XS Importance Matrix 15.7 0.01800 94.7% 48.20% 6.48 66.7% ($p=0.27$) — +2.32 ± 1.80 27.4 tok/s UD-Q3_K_XL Unsloth Dynamic 14.8 0.03800 92.5% 48.16% 6.51 72.5% ($p=0.81$) 31.5% ($p=0.24$) +0.46 ± 1.87 28.7 tok/s Q3_K_M Standard K 13.8 0.05000 91.0% 48.10% 6.50 73.3% ($p=0.65$) — +0.80 ± 0.76 32.1 tok/s Q3_K_S Standard K 12.6 0.08300 88.3% 47.49% 6.71 54.2% ($p<0.001$) — +0.99 ± 1.88 31.3 tok/s UD-IQ3_XXS Unsloth Dynamic 12.2 0.08600 88.2% 47.02% 6.82 52.5% ($p<0.001$) — -0.32 ± 2.66 26.0 tok/s UD-Q2_K_XL Unsloth Dynamic 12.0 0.12000 86.5% 45.73% 6.84 56.7% ($p<0.001$) 32.6% ($p=0.30$) -1.25 ± 2.50 30.9 tok/s UD-IQ2_M Unsloth Dynamic 11.0 0.13000 85.5% 45.82% 7.03 54.2% ($p<0.001$) — -0.61 ± 2.13 26.6 tok/s UD-IQ2_XXS Unsloth Dynamic 9.6 0.28000 78.7% 43.53% 8.19 32.5% ($p<0.001$) — -3.29 ± 1.95 27.8 tok/s Metric Definitions and Key Insights

• Statistical Collapse Threshold ($\text{KLD} \le 0.050$ vs. $\text{KLD} > 0.080$): • Quantizations maintaining $\text{KLD} \le 0.050$ ( Q4_K_M , UD-Q4_K_XL , Q3_K_M , UD-Q3_K_XL ) exhibit no statistically significant degradation on complex reasoning (AIME-120 accuracy matches BF16 within s…

Read original ↗ Content from Reddit r/LocalLLaMA（Community）