About

RECLAIM is 100 NeurIPS 2025 papers. Each paper carries one target claim with a fixed tolerance, a metered compute budget in H100 hours, and a tier set by what the authors released. An agent attempts each paper once per tier sweep.

The compute unit throughout is the H100 hour, written H100·h. One H100·h is one hour of one accelerator at the reference rate. Budgets, spend and the predicted ceiling on every chart use that same unit, so a curve that crosses the dashed indigo line has spent more than the paper was predicted to need.

• Run. code, data and weights released; execute or evaluate
• Retrain. code and data released, no released weights; train before evaluating
• Reimplement. no released code; rebuild the method from the paper

Transcripts are shown verbatim. Reasoning, assistant turns, commands, standard output and standard error are the text the run actually produced. Sandbox paths and identifiers are replaced by bracketed tokens such as [user], [host] or [redacted], and each run is named by its agent, its tier and its paper.

The curve above a transcript is cumulative spend across the rounds, raced against the predicted ceiling. The rail beside the rounds is the same curve turned on its side: click a marker to jump to that round. Round cards are collapsed apart from the last two and the final round, which carries the agent's own report.

A pinned auditor, Claude Sonnet 5, reads the run's evidence bundle and its transcript and assigns a score from 0 to 10 with a written rationale. The verdict is derived from that score in code.

• 8 to 10 · reproduced. The claim was met at the stated bar.
• 6 to 7 · partial. A real number came out of the pipeline and missed the claim's bar or its scope.
• 1 to 5 · not reproduced. A score of 1 reads unverifiable when execution was not verified.
• 0 · disqualified. A high severity integrity flag caps the score here.

The auditor also raises flags. A flag names the kind of integrity problem it found, a severity, and the evidence in the transcript that supports it. Rationales, flags and dissection notes are shown as recorded, except that references to internal rubric bands, criterion codes, file names and scheduling terms are replaced by plain words or removed.

A separate dissection of the transcript assigns each run one primary failure mode from a fixed vocabulary. It never changes a grade. The mode is what the overview's stacked bars count, and the run page prints its definition beside it.

Three verdict families colour the whole site: ●reproduced, ◐miss for a partial result, and ✕fault for a run that did not reach the claim.