Qwen3.6-35B decodes ~11.2 B active per pass

The studio’s RTX 5090D measurement — 1.34 TB/s stated bandwidth, ~25% below the 5090 in BW-bound decode — lets us back out that Unsloth’s reported ~320 tok/s for Qwen3.6-35B at NVFP4 on RTX 5090 implies an effective active-weight footprint of ~11.2 B parameters per forward pass. About one-third of nominal. That ratio fingerprints MoE routing where most experts stay dark during decode. The 1.4–2.2× MTP multiplier Unsloth claims for Qwen3.6 stacks on top of that already-lightened footprint, keeping the product inside the bandwidth ceiling — the same stacking breaks for a dense 35B model.

What Unsloth reports

Unsloth’s headline is “train and RL 500+ models up to 2× faster with 70% less VRAM; MoE up to 12× faster.” Every figure carries implicit conditions; few are disclosed uniformly across the README. The table below assembles what is stated alongside each number:

Claim	Figure	Condition attached	Scope
General training/RL speedup	up to 2×	No baseline stated; LoRA/QLoRA implied by context	500+ models
MoE training speedup	up to 12×	DeepSeek, GLM, Qwen, gpt-oss named; no hardware given	MoE family
MoE VRAM reduction	35% less	Co-stated with 12× speed claim	MoE family
RL VRAM reduction	80% less	GRPO, FP8, vision RL named; 7× context also claimed in same sentence	RL paths only
Long-context training speed	3× faster	Packing technique; 500K+ context	Long-context only
Long-context VRAM reduction	30% less	Same packing context as 3× speed	Long-context only
20B at >500K context	fits 80 GB GPU	Single GPU; precision not stated	Specific size/context
Qwen3.6 MTP inference	1.4–2.2×	MTP heads; NVFP4 for “supported GPUs”	Qwen3.6 only
Embedding fine-tuning	~1.8–3.3×	Approximate marker (”~”) given explicitly	Embedding models
DiffusionGemma inference	1.8×	Studio UI only, not standalone library path	Studio path
gpt-oss 20B fine-tuning	2× / 70% less	Colab notebook; conditions inherit from Colab environment	Dense MoE fine-tuning
Gemma 4 E2B fine-tuning	1.5× / 50% less	Colab notebook	Vision fine-tuning

Unsloth reports a dual-license: core package Apache 2.0, Studio AGPL-3.0. That boundary matters for any product that embeds the inference server or the UI.

Analysis

Active-parameter footprint from measured decode rate. For bandwidth-bound autoregressive decode, throughput is governed by:

$$\text{tok/s} = \frac{BW}{\Phi_{\text{active}} \cdot b}$$

where $BW$ is peak memory bandwidth, $\Phi_{\text{active}}$ is the active parameter count, and $b$ is bytes per parameter. Rearranging to isolate $\Phi_{\text{active}}$:

$$\Phi_{\text{active}} = \frac{BW}{\text{tok/s} \cdot b}$$

From the studio’s RTX 5090 decode field note: the 5090D delivers 1.34 TB/s stated bandwidth and sits ~25% below the 5090 in BW-bound decode. Treating that performance gap as bandwidth-proportional gives an implied 5090 ceiling:

$$BW_{\text{5090}} = \frac{1.34 \text{ TB/s}}{1 - 0.25} = \frac{1.34}{0.75} = 1.787 \text{ TB/s}$$

The same field note records imp at ~320 tok/s on Qwen3.6-35B with a native NVFP4 checkpoint on RTX 5090. NVFP4 is 4-bit, so $b = 0.5$ bytes per parameter. Substituting:

$$\Phi_{\text{active}} = \frac{1{,}787 \text{ GB/s}}{320 \text{ tok/s} \times 0.5 \text{ GB/B param}} = \frac{1{,}787}{160} \approx 11.2 \text{ B params}$$

The predicted pass rate at this footprint is $1{,}787 / (11.2 \times 0.5) = 319$ tok/s — within rounding of the measured 320 tok/s. The bandwidth-bound model is tight here. The ratio $11.2 / 35 \approx 32\%$ is consistent with sparse MoE dispatch; Unsloth groups Qwen among the MoE targets for its 12× training claim, placing it in the same architectural class. The Q8_0/Q6K → NVFP4 caching route yields 267–272 tok/s on the same hardware, running 15–17% below the native NVFP4 checkpoint — the throughput cost of on-the-fly format conversion. Use the pre-quantized checkpoint where the GPU supports it.

MTP multiplier and the bandwidth ceiling. Unsloth reports 1.4–2.2× inference improvement for Qwen3.6 with MTP. Multi-token prediction via speculative draft heads does not reduce bandwidth consumed per main-model forward pass — it raises effective token throughput by emitting $\bar{k}$ accepted tokens per pass:

$$\text{tok/s}_{\text{eff}} = \bar{k} \cdot \text{tok/s}_{\text{base}}$$

Unsloth’s 1.4–2.2× claim therefore implies $\bar{k} \in [1.4,\, 2.2]$ accepted tokens per main-model pass on their test conditions. Applied to the 320 tok/s baseline, effective output spans $1.4 \times 320 = 448$ to $2.2 \times 320 = 704$ tok/s. Those multiples stay inside the bandwidth envelope because the pass itself still costs only $11.2 \times 0.5 = 5.6$ GB of weight reads. A dense 35B model at NVFP4 would load $35 \times 0.5 = 17.5$ GB per pass, with a theoretical ceiling of $1{,}787 / 17{,}500 \approx 102$ tok/s before MTP is applied — the whole construction depends on sparse active weight load.

The 500K context on 80 GB. Unsloth states that training a 20B model at >500K context fits a single 80 GB GPU with its packing technique, claiming 30% less VRAM versus baseline. Working backwards: if 80 GB is the post-Unsloth budget and the saving is 30%, the naive baseline is $80 / 0.70 \approx 114$ GB — which normally requires two A100/H100 80 GB cards without packing. For studios running single-GPU training nodes, collapsing that to one card is the operationally relevant outcome, whatever the baseline implementation.

What the numbers do not settle

Every Unsloth speedup figure omits comparison baseline, hardware, batch size, sequence length, and whether full fine-tuning or QLoRA/LoRA was used. The 2× general claim and the 12× MoE claim likely measure different techniques against different stock implementations. Neither is reproducible from the README alone. Unsloth’s own notebook table — Gemma 4 E2B, Qwen3.5-4B, gpt-oss 20B — comes closest to stated conditions, but Colab GPU provisioning variance makes exact replication unreliable. The studio has not run any of these training benchmarks.

What this means for itria

Itria ships llama.cpp with Metal acceleration; Unsloth’s training stack does not touch the binary. The relevant overlap is inference-side: how bandwidth governs throughput across hardware tiers. The studio’s edge-device field note shows a 3B Q4 model consuming $220 \times 1.8 = 396$ GB/s on M4 Max against a 400 GB/s rated bus. The bandwidth-bound model holds from Raspberry Pi to server GPU, and the 15–17% gap between a pre-quantized NVFP4 checkpoint and an on-the-fly conversion path is proportionally significant at every tier.

The imp backend’s 42–48% advantage over llama.cpp on RTX 5090 is a hardware-specific kernel optimization; it does not transfer to Metal. On Apple Silicon, decode rate tracks the unified memory bus, not the backend. The Qwen3.6 MTP mechanism is implementable via llama.cpp’s existing speculative decode path, and the implied $\bar{k} \approx 1.4$–2.2 from Unsloth’s figure is a reasonable prior for deciding whether that integration overhead is worth it for a given on-device model size.

GLM-5.2 (744B parameters, 1M-context window, as reported by Unsloth) is GGUF-inference-only at any quantization aggressive enough to fit consumer hardware; at the bandwidths available to itria’s target devices it is not viable. The AGPL-3.0 clause on Unsloth Studio also rules out embedding its inference server in a shipped app without a commercial license.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of unslothai/unsloth and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (15 claim(s) and 25 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (16 derived from it, 10 row(s) supplied from our own tables). The studio has not re-run unslothai/unsloth’s benchmarks; figures attributed to it are its own.

FAQ

Does the 12× MoE training speedup apply to full fine-tuning or only LoRA?

Unsloth's README does not separate these cases. The 12× claim appears in the context of general training improvements, with DeepSeek, GLM, Qwen, and gpt-oss named as targets — all used in practice as fine-tuning bases, not pretraining runs. Applying the figure to pretraining MoE from scratch is not supported by the available documentation.

Can Qwen3.6 MTP be used with llama.cpp's speculative decode path?

MTP uses auxiliary prediction heads trained into the main checkpoint, not a separate smaller draft model. Whether llama.cpp's speculative decode path can consume those heads turns on GGUF export and sampler logic alignment. The accept-rate range implied by Unsloth's claim (k = 1.4–2.2) is a useful calibration point, but the mechanism may require integration work beyond what llama.cpp provides out of the box.

Is the 80% VRAM reduction for RL additive with the 70% reduction for general training?

No — these are claims for distinct training configurations with different memory profiles: RL with GRPO holds a reference model copy and reward buffers that SFT does not. Unsloth does not claim these compound. Treating them as additive would overstate the available reduction for a combined SFT + RL workflow.

What does the AGPL-3.0 on Unsloth Studio mean for a shipped product?

Unsloth reports that the core training library is Apache 2.0 but the Studio UI is AGPL-3.0. The library API is Apache-compatible for commercial use. Wrapping or embedding the Studio server — including its /v1/chat/completions endpoint — in a shipped product requires either full AGPL-3.0 compliance (source disclosure for the combined work) or a separate commercial license from Unsloth.