Tiel-Coder's gaps turn on unstated precision

The numbers in Tiel-Coder’s README span two quantization regimes, two chat templates, and at least one unstated precision. Some of the pairs the README places side by side are directly comparable; others are not. The README’s own framing occasionally treats them as equivalent.

The figures and their conditions

Metric	Tiel-Coder-35B-A3B	Nail	Ornith-1.5	Stock Qwen3.6-35B-A3B	Qwen3.8-27B	Dirk
SWE-bench-Live (solved / 25)	12	—	8 (implied: 12 − 4)	8	16	15
SWE-bench-Live time (min)	8.6 median, 12.3 mean	7.2 median, 15.7 mean	—	5.5	50.2	20.1
MMLU-Pro	73.7	84.0	78.0	85.3	—	—
MMLU-Pro quantization	4-bit (Q4_K_XL)	4-bit	not stated	not stated	—	—
Claw-Eval multi-turn	67.2	60.5	65.3	—	—	—
Claw-Eval conversations	114	114	114	—	—	—
Chat template	Sharp	not stated	not stated	not stated	—	—

All SWE-bench-Live figures are one run per problem. MMLU-Pro uses three seeds. The README does not state the serving hardware, context length, or KV precision for any of these runs.

Where the comparison holds

The cleanest pair is Tiel versus Nail on MMLU-Pro: 73.7 against 84.0, both explicitly at 4-bit. The 10.3-point gap is a real difference between two 4-bit builds of (presumably) different models under the same quantization regime. The README attributes part of Tiel’s deficit to the Sharp template producing shorter answers, which helps exact-match scoring on some questions but costs others.

Claw-Eval multi-turn is internally consistent: 114 scored conversations each, same benchmark. Tiel at 67.2, Ornith-1.5 at 65.3, Nail at 60.5. The 3.8-point answer-quality gain and 5.1-point clarifying-question loss are sub-scores from that same evaluation.

On SWE-bench-Live, the solve counts hold up in a narrow sense: same 25 problems, one run each, same benchmark version. Tiel solves 12, stock Qwen3.6-35B-A3B solves 8, Dirk solves 15, Qwen3.8-27B solves 16.

Reported by peculiar-ragdoll/Tiel-Coder-35B… points· scroll →

Conditions around the comparison

scroll →

12 of 25

SWE-bench-Live problems fixed

8.6 minutes

median time

67.2

Claw-Eval multi-turn score

114

scored conversations each

Where it stops holding

The SWE-bench-Live time figures are not comparable across rows. Tiel’s 8.6-minute median is a 4-bit MoE with 3B active parameters. Qwen3.8-27B’s 50.2 minutes is a dense 27B model. Stock Qwen3.6-35B-A3B’s 5.5 minutes is a different MoE, and the README does not state its precision or serving configuration. A 4-bit MoE and a full-precision dense model have different memory-bandwidth profiles, different KV cache sizes, different token-generation costs — even at the same active parameter count. The README’s “at 2.5× its speed” claim for Dirk versus Qwen3.8-27B is an architecture comparison (MoE vs dense at similar total parameters), not a controlled speed test.

Ornith-1.5’s MMLU-Pro score (78.0) and stock Qwen3.6-35B-A3B’s (85.3) carry no stated quantization. The README places them in the same sentence as the 4-bit Tiel and Nail numbers, which invites a direct subtraction. But 78.0 − 73.7 = 4.3, and the README attributes all 4.3 points to the Sharp template. That attribution is only valid if Ornith-1.5’s 78.0 was measured at the same effective precision as Tiel’s 73.7. If it was measured at full precision or a higher-precision quant, part of that gap is quantization loss and part is the template. The clean attribution is an artefact of the unstated condition. The OBLITERATUS MMLU post hit the same issue: the stated precision of the baseline changed the size of the apparent gap.

Claw-Eval has the same problem. The README does not state whether Nail’s 60.5 or Ornith-1.5’s 65.3 were measured at 4-bit or at full precision. If they were at full precision and Tiel’s 67.2 is at 4-bit, Tiel is outperforming its base by more than the stated 3.8 points once quantization loss is accounted for. If they were also at 4-bit, the 3.8-point gain is purely from the template and imatrix. The README does not resolve this.

What cannot be concluded

The README’s headline claim — “at the rate (and speed, with the right GPU) of Opus 4.6 medium” — cannot be verified from the numbers in the document. Opus 4.6 medium does not appear in any table. The 12/25 solve count is matched against Opus 4.6 medium by assertion, not by a shared benchmark run. Without knowing Opus 4.6 medium’s SWE-bench-Live conditions (which problems, how many runs, what agentic harness, what context length), the equivalence is a claim, not a measurement.

---

Method: this note was drafted by qwen/qwen3.8-27b from a single source — the model card for peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF. Before publication an automated gate re-checked every extracted claim against the source document (116 claim(s) and 70 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (1 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF’s benchmarks; figures attributed to it are its own.

FAQ

Can I run the Q4_K_XL tier on a 24 GB GPU with useful context?

The README says Q4_K_XL is 22.4 GB and "snug on 24 GB." The KV cache for 262k context at 16-bit precision is under 5 GB RAM, but that is the KV portion only — model weights are not included. On a 24 GB GPU, 22.4 GB of weights leaves roughly 1.6 GB for KV and activation. The README implicitly acknowledges this by calling it "snug." If you need the 131k–262k context the README recommends for agentic coding, you will likely need 32 GB. The LLM GPU Memory Calculator can give you a more precise number if you plug in your target context length and KV precision.

Does the missing MTP head affect the benchmark numbers?

The README states that the tiers in this repo do not carry the MTP head, that it was untrained when the tiers were baked, and that removing it "changed no output." The benchmark numbers are therefore from a model without the MTP head. If Ornith later ships a trained MTP head (kurtosis 25.1, a 98-sigma outlier, 825 on the nextn projection) and it changes generation behaviour, the benchmark numbers would not transfer to that variant. The MTP head adds 0.9 GB and is in a separate repo.

Is the 4.3-point MMLU-Pro gap really all template?

Only if Ornith-1.5's 78.0 was measured under the same precision conditions as Tiel's 73.7. The README does not state Ornith-1.5's quantization for that figure. If it was measured at full precision or Q8, quantization loss accounts for some of the 4.3 points and the template accounts for the rest. The README's attribution is internally consistent with its own framing (a quantized build with a custom imatrix that recovers most of the precision gap, leaving the template as the residual), but it is not independently verifiable from the document.

Why does Tiel beat Nail on conversation but lose by 10.3 points on MMLU-Pro?

The README's explanation: the Sharp template produces shorter answers, which helps multi-turn conversation (fewer irrelevant tokens, more direct responses) but hurts MMLU-Pro where some questions reward longer, more exhaustive answers. The 6.7-point conversation advantage and the 10.3-point MMLU-Pro deficit are both consistent with a single trade: answer length. Whether that trade is the right one depends on your workload. For agentic coding, where the README positions Tiel, shorter and more direct is presumably better.