3.5-bit Qwen3.8-Flash-Next on two RTX 3090s beats BF16 on MMLU-Pro

A community GGUF quantization of Qwen3.8-Flash-Next (180B, multimodal) at a 3.5-bit target scores 2.85 percentage points above its BF16 reference on MMLU-Pro, with a McNemar p-value of 0.0047752. The quantized model also runs on 2× RTX 3090s. The BF16 reference does not fit in 48 GB of VRAM at all.

Shape of the thing

The repository (pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF) ships three files:

Component	Size	Bits per weight	Condition
Main weights (GSQ-RCO-3.5bit.gguf)	47.94 GB	3.5 target	Requires embedding shard
Token + n-gram embeddings	103.68 GB	16 (BF16)	Required, kept in source precision
Vision projector (mmproj-BF16.gguf)	0.91 GB	16.17	Vision encoder + projector

Total download is approximately 152.5 GB. License: qwen-community-1.0. Pipeline tag: image-text-to-text. Runtime: llama.cpp pinned to upstream commit f3f1a8f, with Qwen4Exp and Q2_0 support.

The quantization is non-uniform. GSQ trains per-tensor candidates for routed expert projections (8 steps, 64 training / 4 validation sequences of 1024 tokens), then RCO search assigns each tensor its type (8 steps, 2 Gumbel samples, 8 training / 4 validation sequences, selected by validation teacher KL). Gate/up candidates are Q2_0/Q2_K/Q3_K/Q4_K; down candidates are Q2_0/Q4_0; non-expert matrices use Q8_0. The repository describes this as “a reduced-budget adaptation of GSQ/RCO to native GGUF formats.”

For 2× 24 GB cards the runtime uses --tensor-split to balance weights across GPUs, --override-tensor to pin both BF16 embedding matrices to CPU (backed by disk via mmap), and --no-kv-offload to place the KV cache in system RAM. Context length: 4096 tokens. Parallelism: 1 sequence.

What the numbers say, under which conditions

All figures below are reported by the repository author. The studio has not independently reproduced them.

Metric	GSQ-RCO 3.5-bit	BF16 reference	Condition
MMLU-Pro accuracy	58.25% (1,165/2,000)	55.40% (1,108/2,000)	Zero-shot, no reasoning, 2,048-token context limit, log-prob of single-token answers A–J, no chat template
Paired MMLU-Pro difference	+2.85 pp	—	McNemar p = 0.0047752; 226 quant-only / 169 BF16-only correct
Native perplexity	3.1058	3.0533	Eight 1,024-token contexts, 4,088 scored tokens, document-disjoint held-out text, greedy decoding
Approximate KL/token	0.111225	—	Same perplexity protocol
IFEval strict	13/16	13/16	Answered directly, 1,024/2,048-token cap
IFEval completed correct	12/16	13/16	One quant response was truncated
GSM8K completed correct	8/8	8/8	xhigh thinking, 1,024/2,048-token cap

The MMLU-Pro protocol is a log-probability scoring task. The model sees a zero-shot prompt; the score is the log probability of the correct single-token answer. No generation, no chain-of-thought, no chat template. The 2,048-token context limit means the model is not asked to reason over long passages. Both quant and BF16 use matching layer placement, so the comparison is not confounded by offloading differences.

The perplexity gap is 0.0525 in absolute terms, which the repository states as 1.72% higher. Over 4,088 scored tokens that is small. It is in the expected direction for a 3.5-bit quantization.

The IFEval drop from 13/16 to 12/16 on completed-correct comes from one truncated response. GSM8K is 8/8 for both — not a useful discriminator at that sample size.

Reported by pfeifferj/Qwen3.8-Flash-Next-GS… GB· scroll →

Conditions around the comparison

scroll →

3.5

target bits

0.91 GB

size of vision

58.25%

MMLU-Pro accuracy

55.40%

MMLU-Pro accuracy of bfloat16 reference

What this is actually useful for

A 180B-parameter multimodal model on two 3090s. That is the pitch. The 48 GB VRAM ceiling is the same constraint that showed up in our earlier note on VRAM-laced quantization, where even 7B Q4 models exceeded 8 GB once KV cache and overhead were accounted for. Here the strategy is more aggressive: the embeddings (103.68 GB) never touch VRAM. They are mmap’d from disk; the GPU only sees the 47.94 GB of quantized transformer weights split across two cards.

Two 24 GB GPUs, a fast NVMe, 32+ GB of system RAM. That is the hardware floor. The llama.cpp commit is pinned, so the runtime is reproducible.

What this is not:

• A throughput story. No tokens-per-second or latency figures are reported. With embeddings on disk and KV cache in RAM, the bottleneck is almost certainly memory bandwidth, not compute. Expect this to be slow by any standard that matters for interactive use.
• A general-purpose quality match to BF16. The MMLU-Pro advantage is measured under a narrow protocol (zero-shot log-prob, no reasoning, 2048 context). The perplexity degradation, though small, is real. The IFEval truncation suggests generation quality under longer outputs may be more fragile than the aggregate scores indicate.
• A drop-in for production serving. Single sequence, 4096 context, no batching configuration documented. The --no-kv-offload flag and mmap embedding access are fine for a single user on a workstation. They are not a serving architecture.

What cannot be concluded from the repository

A quantized model outperforming its full-precision reference on a log-probability scoring task is not the same as being “better” in any generative sense. The protocol rewards calibration of single-token probability mass, which can shift under quantization in ways that do not transfer to open-ended generation or multi-step reasoning. The 1.72% perplexity increase is the more honest signal of information loss, and it is in the direction you would expect. Without a generation-quality evaluation — side-by-side on open-ended prompts with reasoning enabled, at longer context — the MMLU-Pro delta should be read as a protocol artifact rather than a quality claim.

---

Method: this note was drafted by qwen/qwen3.8-27b from a single source — the model card for pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF. Before publication an automated gate re-checked every extracted claim against the source document (75 claim(s) and 45 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (5 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run pfeifferj/Qwen3.8-Flash-Next-GSQ-RCO-GGUF’s benchmarks; figures attributed to it are its own.

FAQ

Do I need the full 152.5 GB, or can I skip the embedding shard?

No. The main weights file explicitly states it "requires embedding shard." The 103.68 GB BF16 embedding file is mandatory and is what gets mmap'd from disk during inference. Without it, the model cannot map token IDs to representations. There is no partial-precision embedding option in this release.

Why is a 3.5-bit quantization of a 180B model only 47.94 GB of weights?

The 3.5-bit target applies to the transformer weight matrices (attention, MLP, expert projections). Token and n-gram embeddings are retained at full BF16 precision (16 bits per weight) and live in the separate 103.68 GB shard. The 47.94 GB figure covers only the quantized transformer parameters. At 3.5 bits per weight, 180B parameters would be roughly 78.75 GB if all parameters were quantized at that rate; the actual 47.94 GB reflects that the non-expert matrices are kept at Q8_0 and the embeddings are excluded from this file entirely.

Is the llama.cpp commit going to stay compatible, or do I need to pin it?

The repository pins to commit f3f1a8f and states it requires Qwen4Exp and Q2_0 support. If upstream llama.cpp changes the GGUF format, adds new tensor types, or alters how mmap'd CPU tensors are handled, this build may break. Pinning the commit is the safe path. The --override-tensor flags and --no-kv-offload behavior are specific to that revision's CLI surface.

Can I use this for vision tasks, or is the text-only path the reliable one?

The vision projector file (0.91 GB, 16.17 bpw) is included and the server command includes --mmproj, so the multimodal path is supported. No vision-specific benchmarks are reported in the repository. The quality of image-conditioned generation is unverified by the author's own numbers. The text-only path is the one with evaluation data behind it.