Three Budget Raises and the Wrong Lever

Here is a question with an obvious answer that is wrong. Something you are running keeps getting cut off before it finishes. What do you do? You give it more room. Anyone would. We did it three times, carefully, measuring each time — and the cut-off rate did not move by a single point. The thing that fixed it was a second knob that pointed in the opposite direction: we asked the model to think less hard, and it started finishing. Quality held. The bill halved. This part is about why the obvious lever was the wrong one, and about the small discipline that turned a configuration change into a result we can stand behind.

The symptom, and the reflex

The machine’s job here is to reason its way through a physics question and hand back claims the rest of the pipeline can check. Roughly half the time, it did not hand back reasoning at all. The thinking block came back empty — started, cut off, gone. 51% of them.

The reading is natural: it ran out of room. So we gave it more room. The thinking budget went up. It went up again. It went up a third time, ending more than four times where it started — from 4,500 to 20,000.

The truncation rate stayed at 51%.

That is the part worth sitting with. Not “it improved a little” or “we were still tuning”. Three deliberate increases, each one measured, and the number stood exactly still. A budget you raise four-fold that changes nothing is not an underpowered budget. It is telling you it was never the thing in the way.

The other knob

There were two controls on this model, not one, and they are easy to conflate. The budget says how much room the reasoning may use. A separate setting — effort — says how hard to push before answering at all. They sound like the same dial at different resolutions. They are not.

So instead of a fourth raise, we went the other way and turned the effort setting down one notch, from its highest level to the middle one. Nothing else changed. Same question, word for word. Same pipeline, same models throughout.

Truncation went from 51% to zero. Not “improved” — zero cut-offs in the whole run, one warning total.

And the things you brace for did not happen:

	before	after
truncated reasoning	51%	0%
answer key	4/4	4/4
cost	$10.71	$4.63
wall clock	65 min	25 min
tokens	1.34 M	0.80 M

The quality metric is the one that matters and it is the one that did not move. Four out of four on a sealed answer key before, four out of four after — and the lower-effort run named each of the corresponding physical quantities explicitly rather than leaving the reader to infer them.

So: three rounds of raising a budget bought nothing, and one notch in the other direction removed the problem, cut the wall clock by more than half and took $6.08 off a single run. The right lever had been sitting in the documentation the whole time.

Three changes on the left, one on the right. The column that moved everything is the one nobody reached for — and the quality metric is the one that stayed put.

Why this is a result and not an anecdote

Everything above would be a nice story and nothing more, except for one habit: the predictions were written down and locked before the run.

Seven of them, each as a point estimate and an interval — cost, duration, tokens, truncation rate, answer-key score, how many wrong physics claims we expected to slip through, and how many crystals the run would yield: verified claims that survive the pipeline’s gates and are added to the archive permanently. Crystals are the unit the whole system exists to produce, which is why the cost figures below are quoted per crystal rather than per run. Along with them, the decision rule: if the answer key holds at 3/4 or better and the errors do not increase, the lower setting stays in production.

All seven landed inside their intervals. One — the crystal count — hit its point estimate dead on: six predicted, six delivered.

That sealed sheet is what separates this from tuning. Without it, “we changed a setting and things got better” is an anecdote, and a cheaper, faster run is exactly the kind of result a person talks themselves into. With it, the claim has a shape that could have failed: had the answer key dropped to 2/4, the rule we had already signed said the change loses, however pretty the invoice looked.

MATH BOX

A constraint you raise and nothing happens is not the binding one

The failure mode has a precise name in optimisation. In any constrained problem, a constraint is either binding — the solution sits against it, and relaxing it moves the answer — or slack, in which case relaxing it changes nothing at all. The shadow price of a slack constraint is exactly zero (Dantzig, 1963; the complementary-slackness half of the Karush–Kuhn–Tucker conditions). Three raises with an unchanged outcome are not weak evidence that the budget was slack. Within measurement noise they are a direct reading of its shadow price, and it read zero every time.

The same fact is more familiar in its computing dress. Amdahl’s law says an improvement applied to a fraction p of the work is capped at 1 / (1 − p + p/s) no matter how large the speed-up s — so when p ≈ 0, buying an enormous s buys you nothing (Amdahl, 1967). Both statements say one thing: before you scale a resource, establish that it is the one you are up against. Neither is exotic. Both are routinely skipped, because raising a limit feels like doing something and measuring whether it was the limit feels like delay.

The economics, for the record: $1.53 per verified crystal before, $0.77 after — for the same question, at unchanged answer-key quality. The full cost-per-unit series and the token accounting are the whitepaper’s subject, not this blog’s.

The sheet that separates a result from a cheaper invoice. Note what is not on it: nothing was locked for structure, which is why the next section's finding is an observation.

The finding that was not on the list

The honest part of the run is the part we had not thought to predict.

The lower-effort run produced six crystals, and not one of them connected to anything. We forced the comparison rather than waiting for it — every combination of the new claims, fifteen pairs, put in front of the judge. Fifteen neutral verdicts. The run before it, at the higher setting, had produced a real connection and the pipeline had found it on its own.

So we had gained a halved bill and possibly lost the structure that is the entire point.

It goes in the record with a note we would rather not have had to write: no prediction was locked for structure. The seven sealed numbers covered cost, speed, truncation and correctness, and simply did not cover whether the output would still connect to anything. The metric was not part of the experiment, so what it did that day is an observation, not a result — and it gets labelled as one.

Two explanations fit, and this run cannot separate them. The lower setting may genuinely fail to carry a derivation far enough to crystallise. Or the content that day may have been the wrong shape: it leaned into refutation work, and refutations do not form connections at the higher setting either.

NERD BOX

How the ambiguity was actually settled

The discriminating test was already scheduled: a differently-shaped question — a five-step chain in which each step derives from the one before, so no step can import an outside parameter — run at the same lowered setting. If structure came back, the content profile was the explanation and the cheaper setting was safe. If it did not, the effort suspicion stood.

It came back: two chain edges, inside the predicted two-to-four.

With a caveat that turned out to matter more than the answer. Those two edges were visible to one judge and invisible to the other. The same corpus, the same day, the same claims: one judge found the structure, the other found none. Had we used the second judge as the meter — as we had been, without knowing there were two — the chain question’s own kill condition would have fired, and we would have concluded the cheaper setting destroys structure. It does not. Our meter was blind.

That is a different story with its own part in this series, and it is the reason this one ends with a caveat rather than a clean win: the run that looked like it lost structure was, at least in part, a run we could not see properly.

What survives

Four things, none of them specific to this model or this setting:

Raising a limit that does not move the outcome is a measurement, not a failed attempt. It has told you the limit is slack. Three raises with a flat result is the cheapest diagnostic anyone will ever hand you — if you were counting.
Look for the second knob. Systems that expose an amount usually also expose an intensity, and reaching for the one you know is not the same as reaching for the one in the way.
A configuration change becomes an experiment the moment the answer key precedes it. The cost of writing seven numbers down beforehand is ten minutes. It is the whole difference between a result and a story about a cheaper invoice.
Preregister the metrics you would be embarrassed to lose, not only the ones you expect to win. Ours covered money, speed and correctness — everything except the property the system exists to produce. That gap is why the structure observation is written here as an open question instead of a finding.

The cheaper setting stayed in production. It is still there.

Sources & further reading

G.M. Amdahl, Validity of the single processor approach to achieving large scale computing capabilities (AFIPS, 1967) — the bound on improving a component that is not the bottleneck.
G.B. Dantzig, Linear Programming and Extensions (Princeton, 1963) — shadow prices, and why a slack constraint’s is zero.
H.W. Kuhn & A.W. Tucker, Nonlinear Programming (Berkeley Symposium, 1951) — complementary slackness: the formal statement that a non-binding constraint has no price.
E.M. Goldratt & J. Cox, The Goal (1984) — the same result as an operating discipline: improvements away from the constraint are not improvements.
B.A. Nosek et al., The preregistration revolution (PNAS, 2018) — the register-before-you-run habit these runs are built on.
C.D. Chambers & L. Tzavella, The past, present and future of Registered Reports (Nature Human Behaviour, 2022) — locking the decision rule before the data exists.

— SF3D