qwen3.8-27b looks genuinely impressive on the benchmark tables - beating models many times its size on some of them. but those numbers come from bf16 weights, and nobody here is running a 27b at bf16. we're running the 4-bit at ~17gb because that's what fits on a 4090 or a 24gb mac. so the model that got measured and the model that got downloaded three million times are not the same artifact. what i can't find anywhere: one model, one harness, swept across bf16 / Q8 / Q6_K / Q5_K_M / Q4_K_M / IQ4_XS, with error bars. perplexity charts exist, but ppl can stay nearly flat while something specific - long-context recall, multi-step math, strict tool-call json - quietly falls over. and with a vision model and a 256k window there's a lot more surface area for that to happen than usual. the question i actually care about: is Q4 of the 27b better than a higher-precision smaller model at the same vram? that's the decision everyone makes and i've never seen it measured. yes, red hat/neural magic publish evals for their quants and llama.cpp has kld tooling. nobody does it systematically, at release, on the harness the model card used. does this exist and i've missed it? and if it doesn't - is anyone with spare compute interested? agreeing on a harness and prompt set first is the only hard part; after that it's just a lot of hours. submitted by /u/AuspiciousApple [link] [comments]
阅读原文 ↗ 内容来自 Reddit r/LocalLLaMA（社区）