# 3.5-bit Qwen3.8-Flash-Next on two RTX 3090s beats BF16 on MMLU-Pro A 3.5-bit GGUF quantization of the 180B Qwen3.8-Flash-Next model runs on 2× RTX 3090s and scores 2.85 points above its BF16 reference on MMLU-Pro. Canonical page: https://makerportal.ai/blog/on-device-ai-hackernews-49703818 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-09-15 Section: Readout / On-Device AI · 5 min read Tags: on-device-ai --- A community GGUF quantization of Qwen3.8-Flash-Next (180B, multimodal) at a 3.5-bit target scores 2.85 percentage points *above* its BF16 reference on MMLU-Pro, with a McNemar p-value of 0.0047752. The quantized model also runs on 2× RTX 3090s. The BF16 reference does not fit in 48 GB of VRAM at all. ## Shape of the thing The repository ([pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)) ships three files: | Component | Size | Bits per weight | Condition | |---|---|---|---| | Main weights (`GSQ-RCO-3.5bit.gguf`) | 47.94 GB | 3.5 target | Requires embedding shard | | Token + n-gram embeddings | 103.68 GB | 16 (BF16) | Required, kept in source precision | | Vision projector (`mmproj-BF16.gguf`) | 0.91 GB | 16.17 | Vision encoder + projector | Total download is approximately 152.5 GB. License: `qwen-community-1.0`. Pipeline tag: `image-text-to-text`. Runtime: `llama.cpp` pinned to upstream commit [f3f1a8f](https://github.com/ggml-org/llama.cpp/commit/f3f1a8f2760f28325a5ec20c05b171e5b7), with Qwen4Exp and Q2_0 support. The quantization is non-uniform. GSQ trains per-tensor candidates for routed expert projections (8 steps, 64 training / 4 validation sequences of 1024 tokens), then RCO search assigns each tensor its type (8 steps, 2 Gumbel samples, 8 training / 4 validation sequences, selected by validation teacher KL). Gate/up candidates are Q2_0/Q2_K/Q3_K/Q4_K; down candidates are Q2_0/Q4_0; non-expert matrices use Q8_0. The repository describes this as "a reduced-budget adaptation of GSQ/RCO to native GGUF formats." For 2× 24 GB cards the runtime uses `--tensor-split` to balance weights across GPUs, `--override-tensor` to pin both BF16 embedding matrices to CPU (backed by disk via mmap), and `--no-kv-offload` to place the KV cache in system RAM. Context length: 4096 tokens. Parallelism: 1 sequence. ## What the numbers say, under which conditions All figures below are reported by the repository author. The studio has not independently reproduced them. | Metric | GSQ-RCO 3.5-bit | BF16 reference | Condition | |---|---|---|---| | MMLU-Pro accuracy | 58.25% (1,165/2,000) | 55.40% (1,108/2,000) | Zero-shot, no reasoning, 2,048-token context limit, log-prob of single-token answers A–J, no chat template | | Paired MMLU-Pro difference | +2.85 pp | — | McNemar p = 0.0047752; 226 quant-only / 169 BF16-only correct | | Native perplexity | 3.1058 | 3.0533 | Eight 1,024-token contexts, 4,088 scored tokens, document-disjoint held-out text, greedy decoding | | Approximate KL/token | 0.111225 | — | Same perplexity protocol | | IFEval strict | 13/16 | 13/16 | Answered directly, 1,024/2,048-token cap | | IFEval completed correct | 12/16 | 13/16 | One quant response was truncated | | GSM8K completed correct | 8/8 | 8/8 | xhigh thinking, 1,024/2,048-token cap | The MMLU-Pro protocol is a log-probability scoring task. The model sees a zero-shot prompt; the score is the log probability of the correct single-token answer. No generation, no chain-of-thought, no chat template. The 2,048-token context limit means the model is not asked to reason over long passages. Both quant and BF16 use matching layer placement, so the comparison is not confounded by offloading differences. The perplexity gap is 0.0525 in absolute terms, which the repository states as 1.72% higher. Over 4,088 scored tokens that is small. It is in the expected direction for a 3.5-bit quantization. The IFEval drop from 13/16 to 12/16 on completed-correct comes from one truncated response. GSM8K is 8/8 for both — not a useful discriminator at that sample size.
Reported by pfeifferj/Qwen3.8-Flash-Next-GS… GB· scroll →
size of n-gram 103.68 GB size of main weights 47.94 GB memory 24 GB

Conditions around the comparison

scroll →
3.5
target bits
0.91 GB
size of vision
58.25%
MMLU-Pro accuracy
55.40%
MMLU-Pro accuracy of bfloat16 reference
Figure: values as pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.
## What this is actually useful for A 180B-parameter multimodal model on two 3090s. That is the pitch. The 48 GB VRAM ceiling is the same constraint that showed up in [our earlier note on VRAM-laced quantization](/blog/thumb-dash-hackernews-49279500), where even 7B Q4 models exceeded 8 GB once KV cache and overhead were accounted for. Here the strategy is more aggressive: the embeddings (103.68 GB) never touch VRAM. They are mmap'd from disk; the GPU only sees the 47.94 GB of quantized transformer weights split across two cards. Two 24 GB GPUs, a fast NVMe, 32+ GB of system RAM. That is the hardware floor. The llama.cpp commit is pinned, so the runtime is reproducible. What this is not: - **A throughput story.** No tokens-per-second or latency figures are reported. With embeddings on disk and KV cache in RAM, the bottleneck is almost certainly memory bandwidth, not compute. Expect this to be slow by any standard that matters for interactive use. - **A general-purpose quality match to BF16.** The MMLU-Pro advantage is measured under a narrow protocol (zero-shot log-prob, no reasoning, 2048 context). The perplexity degradation, though small, is real. The IFEval truncation suggests generation quality under longer outputs may be more fragile than the aggregate scores indicate. - **A drop-in for production serving.** Single sequence, 4096 context, no batching configuration documented. The `--no-kv-offload` flag and mmap embedding access are fine for a single user on a workstation. They are not a serving architecture. ## What cannot be concluded from the repository A quantized model outperforming its full-precision reference on a log-probability scoring task is not the same as being "better" in any generative sense. The protocol rewards calibration of single-token probability mass, which can shift under quantization in ways that do not transfer to open-ended generation or multi-step reasoning. The 1.72% perplexity increase is the more honest signal of information loss, and it is in the direction you would expect. Without a generation-quality evaluation — side-by-side on open-ended prompts with reasoning enabled, at longer context — the MMLU-Pro delta should be read as a protocol artifact rather than a quality claim. --- *Method: this note was drafted by qwen/qwen3.8-27b from a single source — the model card for [pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF). Before publication an automated gate re-checked every extracted claim against the source document (75 claim(s) and 45 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (5 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Do I need the full 152.5 GB, or can I skip the embedding shard? No. The main weights file explicitly states it "requires embedding shard." The 103.68 GB BF16 embedding file is mandatory and is what gets mmap'd from disk during inference. Without it, the model cannot map token IDs to representations. There is no partial-precision embedding option in this release. ### Why is a 3.5-bit quantization of a 180B model only 47.94 GB of weights? The 3.5-bit target applies to the transformer weight matrices (attention, MLP, expert projections). Token and n-gram embeddings are retained at full BF16 precision (16 bits per weight) and live in the separate 103.68 GB shard. The 47.94 GB figure covers only the quantized transformer parameters. At 3.5 bits per weight, 180B parameters would be roughly 78.75 GB if all parameters were quantized at that rate; the actual 47.94 GB reflects that the non-expert matrices are kept at Q8_0 and the embeddings are excluded from this file entirely. ### Is the llama.cpp commit going to stay compatible, or do I need to pin it? The repository pins to commit f3f1a8f and states it requires Qwen4Exp and Q2_0 support. If upstream llama.cpp changes the GGUF format, adds new tensor types, or alters how mmap'd CPU tensors are handled, this build may break. Pinning the commit is the safe path. The `--override-tensor` flags and `--no-kv-offload` behavior are specific to that revision's CLI surface. ### Can I use this for vision tasks, or is the text-only path the reliable one? The vision projector file (0.91 GB, 16.17 bpw) is included and the server command includes `--mmproj`, so the multimodal path is supported. No vision-specific benchmarks are reported in the repository. The quality of image-conditioned generation is unverified by the author's own numbers. The text-only path is the one with evaluation data behind it.