We improved the metric.
Then the model stopped answering.
On the confirmatory partition, tool-calling F1 rose from 0.6264 to 0.7470 with SFT; DPO added +0.0078, within noise, and that checkpoint was promoted under a gate revised after SFT was evaluated. A later diagnosis on real IFEval, GSM8K and MMLU-Pro found a regression the gate was structurally unable to see — because its held-out set contained no examples that simply asked the model to answer.
The promoted checkpoint is a research artifact, not a general-purpose improvement. The capability numbers recompute from 48,840 committed per-example records without a GPU; the tool-policy and corpus figures come from the committed evaluation metrics and the published corpus.
The intervention worked.
This is not a story about a training run that failed. Canonical-v2 SFT did what it was designed to do: teach the model when to call a tool rather than how often. Almost all of the gain arrived with SFT; the DPO calibration step on top of it moved the metric by less than this partition can resolve.
The target metric rose. This part is real.
Over-call fell from 62.4% to 15.0% with SFT; DPO left it at 15.3%, slightly higher. Almost all of the F1 gain arrived with SFT — DPO added +0.0078, 7 of 453 calls, from a single seed. The policy genuinely got more selective — that is not what this page disputes.
What the promotion means. M1-v2 is promoted under a parent-relative gate (v4) introduced after M0 was evaluated; M0 also clears v4, and M1-v2 fails the v3 gate that rejected M0. The promotion reflects the gate change; the measured difference from M0 (+0.0078 call_f1, 7 of 453 calls, single seed) is within noise.
A seven-case smoke test said something was wrong.
Running an external sentinel suite, the promoted checkpoint scored 5/7 — the same as the untrained base model. The scores matched. The behaviour did not.
Identical score. Opposite failure.
Base and the promoted checkpoint both score 5/7, but not on the same items. Base gets trap arithmetic and multi-step change wrong; M1-v2 refuses multi-step change and the format constraint, and passes trap arithmetic. A scalar score cannot tell those apart — which is why seven cases never carry a conclusion here.
SUGGESTIVE, not evidence. Seven cases cannot support a capability claim — one case is 14 percentage points. It was enough to justify a real measurement, and nothing more.
Real benchmarks, pinned revisions, deterministic scorers.
No synthetic stand-ins, no approximated subsets presented as official scores, and no LLM judge anywhere. Fourteen benchmark adapters in the repository turned out to synthesise placeholder tasks; those were recorded as blocked rather than scored.
google/IFEvalupstream checkers, vendored unmodified, seeded
openai/gsm8kdeterministic numeric extraction
TIGER-Lab/MMLU-Proupstream answer regexes
All four checkpoints ran on one engine configuration — vLLM 0.29.0, bfloat16, greedy decoding, seed 0, prefix caching off — so no stage difference is attributable to the runtime.
It refuses every bare arithmetic question.
All 1,319 of them. Not a sample — the entire GSM8K test split, with the same handful of phrasings repeated across it.
“Apologies, but I’m unable to perform calculations or solve problems. I recommend using a calculator…”
M1-v2 · 1,319 of 1,319 zero-shot GSM8K responses · mean length 132 characters
| Stage | GSM8K 0-shot | GSM8K refusal | GSM8K 8-shot | IFEval strict | MMLU-Pro | Sentinel |
|---|---|---|---|---|---|---|
| Base | 67.4% | 0% | 70.4% | 67.8% | 49.0% | 5/7 |
| M0 · SFT | 0.0% | 100% | 56.3% | 45.1% | 37.0% | 5/7 |
| M1-v2 · DPO | 0.0% | 100% | 55.5% | 45.8% | 37.0% | 5/7 |
| M1-v1 · DPO on base | 17.3% | 71% | 70.4% | 43.4% | unmeasured | 3/7 |
Is it unable, or unwilling?
These need different fixes, so they cannot be collapsed. The test was registered before any output was generated: run the identical questions again, with eight worked examples in front of them.
Only the exemplars change.
Content is held constant. A model that refuses 100% of bare questions solves 55.5% of the same questions when shown how the answer should look.
Not a subject. A request shape.
Zero refusals on 12,032 MMLU-Pro items — including 1,351 maths questions — from the same checkpoint that declines every bare arithmetic word problem. n/a is unmeasured, not zero: M1-v1’s MMLU-Pro run was terminated on budget.
Refusal is not the whole story.
It would be convenient if the ability were intact and merely unused. It is not. In regimes where the model refuses essentially nothing, it is still materially worse than the base model.
Refusal cannot explain these.
IFEval ran with a 2,560-token cap, and 47 (Base), 67 (M0), 63 (M1-v2) and 17 (M1-v1) of its 541 responses hit it. Every IFEval figure on this page carries that caveat; no interval has been computed for it.
The gap survives the worst admissible case.
Worst case gives Base its floor and M0 its ceiling. These are deterministic ranges, not confidence intervals — no distribution is assumed.
It appears after SFT — and after DPO with no SFT at all.
The natural assumption is that preference optimisation over-aligned the model into refusing. On the promoted lineage the signal appears at SFT and DPO on top of it leaves it flat. But DPO applied directly to the base model, with no SFT, also produces most of the refusal.
A general-capability regression associated with tool-policy post-training on When2Call-derived data. It appears after SFT (M0) and also after DPO applied directly to the base (M1-v1), so it is not specific to SFT. Causation is not established: one lineage, one seed, no replicate. “DPO is flat” describes the M0 → M1-v2 edge on this suite only; it is not a claim about weights, optimiser effectiveness, or what targeted preference training could achieve.
Our first MMLU-Pro result was wrong — in our own favour.
It showed almost no regression. That was a measurement artifact, caught by checking why one checkpoint's answer rate was low rather than accepting the number.
At 768 tokens the regression was invisible.
Looks flat. Would have been reported as “no degradation.”
A 12.0pp gap. The correction strengthened the finding.
The budget bit Base five times harder than M0, so the benchmark was measuring verbosity, not accuracy. 99.6% of Base’s unattempted items at 768 (4,292 of 4,309) were truncations — not refusals.
The superseded run is retained, not deleted, and its $3.89 cost is counted in full. A protocol that would have falsely implied “no degradation” is itself a finding worth keeping.
The gate could not have caught this.
The frozen confirmatory partition behind the promotion decision (n=1,277) held tool-call, clarification and cannot-answer examples. It held no examples that simply asked the model to answer something.
The DEV split used for checkpoint selection and the full 3,650-example held-out set (1,295 / 1,060 / 1,295) have the same gap. A promotion gate is only as broad as the held-out set behind it. Future gates now require coverage of all four response modes and must reject any candidate whose tool policy improves while direct-answer behaviour collapses. Updated promotion policy
The supervision teaches refusal as the way to answer.
A CPU audit of M0's SFT corpus (Canonical-v2 final) found refusal text sitting under the ANSWER label — at scale, and concentrated in exactly the prompt shape that triggers refusal at inference.
Refusals taught as the way to answer.
18,114 records (10.5% of 173,237) carry a refusal as the supervised target — and every one is labelled ANSWER. None is labelled CANNOT_ANSWER. An earlier audit quoted 21,749 of 217,903; that read the normalization-v1 sources, not M0’s corpus. This is an association, not a demonstrated cause; the ablation that would establish causation has not been run.
“I’m sorry, I can’t provide an answer to that. Calculating diabetes risk involves complex factors…”
“Apologies, but I’m unable to perform calculations or solve problems.”
Association, not proof. Template overlap is only 2.9% — the model learned the behaviour and register, not the phrasings. M1-v1 never saw this SFT corpus — it is DPO on When2Call preference pairs, applied directly to the base — and still refuses 70.7% of bare GSM8K, so these labels cannot be the whole explanation. And the defect is the label, not the refusal: many of these refusals are correct, and deleting them would trade over-refusal for hallucination.
Claims we examined and threw out.
The campaign was audited as if by an opponent. These are the readings the evidence does not support — including several we had written ourselves.
It solves 55.5% of GSM8K with exemplars and never refuses MMLU-Pro.
−14.1pp on 8-shot GSM8K where refusal is ~0%.
100% → 0% refusal from exemplars alone, content held constant.
M1-v1 — DPO applied directly to the base, no SFT — refuses 70.7% of bare GSM8K. Causation is not established either way.
That was the 768-token truncation artifact. Corrected: −12.0pp.
Better than Base on tool policy, within noise of M0 (+0.0078 F1, 7 of 453 calls), materially worse than Base on every general measure here.
One case is 14pp. Base and M1-v2 both score 5/7 with opposite failures.
Arithmetic is accessible under 8-shot, at ~14pp below Base. Not intact.
Corrections the audit forced on our own write-ups
| Status | Prior claim | Correction |
|---|---|---|
| FALSE | “only 1 of 2,638 GSM8K generations truncated” | True of M1-v2 alone. Base 178, M1-v1 256. |
| INVALID | “the −12pp drop is a conservative lower bound” | No bound had been computed. Derived: +6.4 / +12.0 / +33.1pp. |
| NON-REPRODUCIBLE | IFEval scores | Scorer was non-deterministic (0.4510 / 0.4492 / 0.4492). Seeded and re-scored. |
| QUALIFIED | “engine change ≈ quantization” | Controlled, but the engine arm also changed hardware. Now a runtime-stack claim. |
| CORRECTED | “$22.70 remaining” | A planning envelope, never a queried balance. |
| WRONG | “SFT introduced it; preference training did nothing” | M1-v1 is DPO directly on the base (no SFT parent) and refuses 70.7% of bare GSM8K. Not specific to SFT; causation not established. |
| MIXED POPULATIONS | “call F1 rose from 0.6191 to 0.7548” | 0.6191 is B0 on the full 3,650 set. Confirmatory: 0.6264 → 0.7470 (SFT) → 0.7548 (DPO, +0.0078, within noise). |
| QUALIFIED | “promoted under a pre-registered gate” | Promoted under v4, a parent-relative gate committed after M0 was evaluated. M0 also clears v4; M1-v2 fails the v3 gate that rejected M0. |
| WRONG CORPUS | “21,749 of 217,903 SFT targets are refusals” | Measured the normalization-v1 sources, not M0’s corpus. Canonical-v2: 18,114 of 173,237 (10.5%); When2Call 4,038 of 6,505. |
A second pass, over every claim. A later audit rechecked about 990 number and status claims in the repository, the Hugging Face cards and this page against the committed artifacts. It found 93 that did not hold, 13 of them high severity. Each one is listed with what the artifacts show and how it was corrected in the claim audit (PDF).
What the campaign cost, and what it preserved.
Superseded work is counted in full and retained. Failed launches are separated from benchmark cost. No account balance is ever asserted.
$22.7 was a planning envelope, never a queried balance. Treating it as spendable led to over-committing against real credit — the ledger now records envelope_is_not_a_balance: true and REMAINING_CREDIT_BALANCE = NOT_QUERYABLE. Engine parity is held separate from the checkpoint study: vLLM ↔ llama.cpp agreement is 0.983555 with 21 decision flips over 1,277 examples.
Compressing the promoted model: 9 formats, 2 that pass.
The promoted checkpoint was converted to GGUF and quantized 9 ways for llama.cpp, each format scored on the same confirmatory examples against a gate frozen before any format existed. The question is not whether the scores move, but whether the decisions do.
Below 1.3 GiB, at least 11.7% of decisions change.
Every format was quantized directly from the BF16 GGUF with the same importance matrix, on the same engine and tokenizer, so these differences are attributable to quantization. A flip is a prompt where the decision changed: at Q6_K, 25 flips split into 10 fixed, 7 broken and 8 sideways.
| Format | Size | Call F1 | Precision | Recall | Over-call | Clarify acc. | Unsupported acc. | Valid output | Frozen gate |
|---|---|---|---|---|---|---|---|---|---|
| vLLM BF16 (frozen reference) | — | 0.7548 | 0.7358 | 0.7748 | 15.3% | 76.5% | 53.9% | 100.0% | reference |
| BF16 GGUF (unquantized) | 3.52 GiB | 0.7572 | 0.7344 | 0.7815 | 15.5% | 76.8% | 54.8% | 100.0% | passes |
| Q2_K | 0.90 GiB | 0.6713 | 0.5548 | 0.8499 | 37.5% | 15.4% | 66.9% | 98.4% | rejected |
| Q3_K_M | 1.02 GiB | 0.7415 | 0.6356 | 0.8896 | 28.0% | 55.0% | 49.2% | 100.0% | rejected |
| IQ3_M | 0.99 GiB | 0.7057 | 0.5783 | 0.9051 | 36.3% | 57.4% | 37.3% | 100.0% | rejected |
| IQ4_XS | 1.11 GiB | 0.7531 | 0.6628 | 0.8720 | 24.4% | 66.3% | 51.9% | 100.0% | rejected |
| Q4_K_S | 1.13 GiB | 0.7578 | 0.6650 | 0.8808 | 24.4% | 64.1% | 51.7% | 99.9% | rejected |
| Q4_K_M | 1.19 GiB | 0.7594 | 0.6612 | 0.8918 | 25.1% | 63.9% | 50.1% | 99.9% | rejected |
| Q5_K_M | 1.31 GiB | 0.7612 | 0.7213 | 0.8057 | 17.1% | 74.7% | 52.1% | 100.0% | rejected |
| Q6_K recommended | 1.45 GiB | 0.7572 | 0.7286 | 0.7881 | 16.1% | 76.5% | 55.0% | 100.0% | passes |
| Q8_0 | 1.87 GiB | 0.7543 | 0.7308 | 0.7792 | 15.8% | 76.5% | 54.5% | 100.0% | passes |
| Gate threshold | — | ≥ 0.7473 | ≥ 0.7285 | ≥ 0.7671 | ≤ 16.3% | ≥ 75.8% | ≥ 53.3% | ≥ 99% | 99% of reference; over-call at most +1 pt |
Red cells fail the frozen gate. The first row is the vLLM reference the gate was built from; the second is the same weights as an unquantized GGUF on llama.cpp.
F1 alone would have shipped the wrong format. Q4_K_M scores a higher call F1 than the unquantized model (0.7594 against 0.7572), yet it calls a tool on 25.1% of the examples that need no call, against 15.5%. The gate checks each dimension separately, so it rejects it.
The pass is inside the noise. Q6_K clears the precision floor by one example (357 correct of 490 predicted calls, against 356.96 required), and a rerun of the unquantized model on an H200 fails the same gate on recall (0.7638 against 0.7671). “Q6_K and Q8_0 pass” is a correct reading of the frozen gate, not evidence that they preserve behaviour better than running the original again would. No format cleared the stricter release bar, so the fallback registered before any format existed applies: the smallest format that passes the gate. That is Q6_K.
- The capability regression. The confirmatory set holds no plain-answer examples, and no quantized format was re-run on GSM8K, IFEval or MMLU-Pro.
- Exact tokenizer parity. llama.cpp tokenizes 6 of 1,277 prompts differently from the source tokenizer. That is identical in every format, so it cancels in the comparisons above.
- Phone or CPU speed, and more than one run per format for the behavioural scores.
On a datacenter GPU, quantization mostly buys memory.
| Format | Size | Generation (tok/s) | vs BF16 | Prompt, 2,048 tokens (tok/s) | vs BF16 |
|---|---|---|---|---|---|
| BF16 GGUF | 3.52 GiB | 251.1 ± 1.9 | — | 21,073 ± 23 | — |
| Q2_K | 0.90 GiB | 284.8 ± 3.6 | +13% | 9,623 ± 6 | −54% |
| Q3_K_M | 1.02 GiB | 268.8 ± 3.6 | +7% | 10,879 ± 12 | −48% |
| IQ3_M | 0.99 GiB | 283.4 ± 3.1 | +13% | 12,420 ± 27 | −41% |
| IQ4_XS | 1.11 GiB | 306.3 ± 3.5 | +22% | 13,288 ± 20 | −37% |
| Q4_K_S | 1.13 GiB | 299.2 ± 5.6 | +19% | 12,436 ± 40 | −41% |
| Q4_K_M | 1.19 GiB | 292.7 ± 3.2 | +17% | 12,200 ± 9 | −42% |
| Q5_K_M | 1.31 GiB | 292.9 ± 1.7 | +17% | 12,414 ± 20 | −41% |
| Q6_K | 1.45 GiB | 267.7 ± 2.8 | +7% | 11,914 ± 18 | −43% |
| Q8_0 | 1.87 GiB | 273.8 ± 1.8 | +9% | 13,297 ± 22 | −37% |
Against BF16, the quantized formats generate 7–22% faster but process prompts at only 46–63% of its rate. These are A100 figures. On a phone or CPU, where memory bandwidth is the limit, the balance can differ, and it was not measured here.
The next experiment tests relabelling, not deletion.
The obvious move — strip the refusal records and retrain — is wrong. Many of those refusals are correct: a model genuinely cannot report today’s weather or query a VIN database. Deleting them would trade over-refusal for hallucination, which is worse and harder to detect.
Relabelling is a hypothesis, not a known fix. It targets the SFT supervision, but DPO applied directly to the base model on the same When2Call data also produced the refusals, so the preference pairs need the same scrutiny. It counts as a fix only if a relabelled run shows the regression shrinking.
- M1-v1 corrected-budget MMLU-Pro — UNMEASURED, not interpolated
- Whether refusal-as-ANSWER supervision caused the behaviour — association only
- Why DPO on the base (M1-v1), with no SFT, also refuses 70.7% of bare GSM8K
- Single lineage, single base model, no replicate
- Whether DPO could repair it if explicitly targeted
- Whether the effect generalises to other model families
- The internal mechanism of the refusal behaviour