ICML 2026 orals: only 8 of 105 survived replication

I review models by running them, not by reading about them. That habit started as stubbornness. It became self-defense. SAI, a research-verification startup, just published the strongest argument for it I've seen: it took every ICML 2026 oral paper, 168 of them, and tried to actually run the experiments. The headline is brutal. Only 8 of the 105 papers it fully replicated held up on more than 80% of their claims.

For China AI watchers, this debate has a familiar shape: open weights made verification a user task.

The numbers

SAI ran the audit in the weeks after the conference (July 6-11, Seoul) and published the report on July 22. ICML 2026 took 23,918 submissions and accepted 6,352; the 168 orals are the top presentation tier, about 0.7% of everything that entered review.

One paper alone reported 38,400 GPU hours on 80 A100s for a single pretraining stage. SAI's cost estimates exclude salaries and infrastructure, and assume one-shot runs; real research retries make them 2-3x higher. A full oral is not a cheap artifact.

What "replicated" means here

SAI's agents extract every claim from the paper, fetch the code, run it, and score the claims. Claims fall into three buckets: not attempted, attempted and ran (possibly reduced scope), and ran fully in scope. The main score uses the middle bucket, for papers with at least five such claims. The conclusion is robust to the choice: using attempted-and-ran claims, under 10% of papers cleared 80%; using fully-in-scope runs, about 20% did. Either way, most of the top tier mostly fails to check out.

Where the papers broke

The failure causes, largest first: code doesn't run as shipped (missing files, underspecification, dependencies agents can't fix), and code runs but produces different results than claimed. Four papers depend on retired models, which means those results "can no longer be reproduced by anyone."

Two examples from the report are invisible if you only read the manuscript:

• A paper's headline selling point is that its method trains only 0.77% of the base model's parameters. The checkpoint it released trains 6.31%, about 8x more. The 0.77% figure only works out for an adapter one eighth the size of the one shipped.
• Another paper reports a reliability table scored against a judge model that appears nowhere in its released code. Nothing in the repo computes those numbers.

Fairness note: a low score doesn't mean deception. Code goes stale, data stays proprietary, machines differ. But the practical consequence is the same. The science in those papers cannot be validated.

The leak that started the week

The audit didn't start the debate; a leak did. On August 4, Mathew Shen posted about an ICLR 2026 paper whose results were suspiciously good. The code was open, so he checked: it was selecting parameters using test-set labels. His closing line, roughly translated: "this is what a top conference looks like?" Chenhao Tan, a UChicago CS professor, answered with four characters: "所以我们做了这个" — so we built this — and linked the audit. The report itself went out on July 22, and Tan posted it to X the same day.

Same day, Keller Jordan, who pretrains at OpenAI, posted a PSA: most people in big labs now read almost zero papers, and understand ICLR/ICML/NeurIPS to be "mainly full of overclaims & fraud," with a few diamonds in the rough. He walked it back a bit the same day: he wouldn't claim a numerical majority of papers are fraudulent; it's more that if you select for the most exciting papers, those are mostly overclaims.

The replies carried the argument forward. Stella Biderman, director of EleutherAI, pushed back on the "big lab" framing: the problem is true for leaders in the field regardless of where they work. Another reply: "most papers feel like theyre just cargo culting research for hiring purposes." And a useful one from Bharath Hariharan: reviewers rarely look at code and assume good faith, but once you've found a leak, you can post a public comment on the paper. The leakage case only surfaced because the code was open. That one anecdote is the whole argument for open code.

What I'd change about how we read research

My take, stated plainly: the manuscript is a hypothesis, not a result.

Three practices I've adopted, in case they help:

1. Start with the repo, not the abstract. If there's no code, or the code won't run, discount the claims accordingly. SAI found 104 of 168 orals shipped runnable code — a third of the top tier shipped nothing runnable.
2. Prefer numbers someone actually re-ran. When a number decides an architecture choice, I want a run I can check, not a figure I can't. This is why this site runs its own batteries and publishes raw logs; the SAI audit is the same instinct applied to papers.
3. When you find a leak, say so in public, on the paper. Reviewers assume good faith. The next reader shouldn't have to rediscover it.

Two caveats to keep the note honest. SAI is a startup selling review services, and the audit doubles as a product demo. To its credit, the report flags that SAI review is still in beta and can make mistakes. And the audit grades reproducibility, not importance — a paper can fail replication and still contain the idea you needed.

The open-weights counterpoint

Which brings us to China. The verification movement's most visible builder is a Chinese scholar: Chenhao Tan's group at UChicago wrote OpenAIReview and VERITAS, the two open-source systems SAI extends. That's worth a beat for anyone who still sees China as the closed-source side of the field.

The counterpoint to closed reproducibility is open weights. DeepSeek, Qwen, GLM, Kimi, MiniMax all ship weights you can download and run. Nobody needs an audit to verify them; you can check the checkpoint yourself. Open weights turned reproducibility from a reviewer problem into a user problem, and China's labs have been the loudest exporters of that model. The audit's answer to "who verifies the papers?" is agents. The open-weights answer is: don't need them.

Bottom line

The audit doesn't prove mass fraud at scale. It proves verification is the missing layer of the ecosystem. Read papers as claims, run what you can, demand code. Papers are proposals. Runs are answers.

Sources

1. SAI (How much science is verifiable?), sai.science/blog/how-much-science-is-verifiable (published 2026-07-22, captured 2026-08-09, first-party audit, beta disclaimer)
2. UCL Engineering (Papers accepted at ICML 2026), ucl.ac.uk/engineering/news/papers-accepted-icml-2026 (captured 2026-08-09, third-party statistics; cross-checked with OpenAccept, openaccept.org/c/ai/icml/2026/)
3. X @MathewShen42 (ICLR 2026 leakage thread), x.com/MathewShen42/status/2084465434506768867 (posted 2026-08-04, first-person account; Chinese original, translated in text; Bharath Hariharan's reply in thread)
4. X @kellerjordan0 (OpenAI pretraining; PSA and same-day walkback), x.com/kellerjordan0/status/2084721463089902074 (posted 2026-08-04, opinion, quoted; Stella Biderman and will depue replies in thread)
5. X @ChenhaoTan (UChicago CS professor; "所以我们做了这个" reply in source 3's thread; audit announcement 2026-07-22), x.com/ChenhaoTan (captured 2026-08-09, self-described)
6. arXiv (Benchmarking Agentic Review Systems — OpenAIReview), arxiv.org/abs/2606.19749 (submitted 2026-06-18, preprint)
7. arXiv (VERITAS), arxiv.org/abs/2607.02931v1 (submitted 2026-07-03, preprint)

Disclosure: no sponsors, no affiliate links. All numbers above are compiled from public sources with capture dates; we did not run the SAI audit ourselves. SAI is a startup selling review services, and the audit doubles as a product demo — the report itself flags that SAI review is still in beta.