Why AI Scientists Don't Do Science: Inside Corral's 25,000-Run Epistemic Audit
A landmark study across 8 scientific domains reveals LLM agents ignore evidence 68% of the time, and scaffolding accounts for only 1.5% of variance over base model reasoning.
A correct scientific result does not prove an AI conducted scientific reasoning. Across thousands of published papers and demo videos, autonomous AI "scientists" have synthesized molecules, designed computational workflows, and calibrated lab instruments. But when researchers from Friedrich Schiller University Jena, the Helmholtz Institute, and IIT Delhi subjected frontier models to an exhaustive 25,000-run audit across eight experimental domains, they discovered that modern LLM agents routinely land on correct answers through statistical luck while flagrantly violating the basic epistemic norms of the scientific method.
The study, titled "AI scientists produce results without reasoning scientifically" and backed by the open-source Corral benchmarking suite, introduces a rigorous way to evaluate not just what an agent outputs, but the epistemological structure of how it gets there.
[Hypothesis H1] ──> [Tool / Experiment] ──> [Contradictory Evidence E1]
│
└──> (68% Evidence Non-Uptake) ──> [Maintains H1 / Guesses Target]
The findings are a stark reality check for autonomous discovery: agents ignore their own experimental evidence in 68% of traces, update their beliefs in response to refutation only 26% of the time, and rely on scaffold engineering that turns out to explain a negligible fraction of real performance.
The Scaffold Fallacy: 41.4% Base Model vs. 1.5% Wrapper
Over the past two years, the AI agent ecosystem has poured vast engineering effort into "scaffolds"—complex ReAct prompt loops, hierarchical planning orchestrators, reflection sub-agents, and dynamic tool-selection harnesses. The implicit assumption was that with the right harness, even standard frontier models could be guided into executing self-correcting scientific loops.
Corral's variance decomposition completely dismantles this assumption:
- Base Model Reasoning Drives Performance: Using latent factor modeling, the researchers found that base model capabilities account for 41.4% of explained variance in agent performance and behavior.
- Scaffold Engineering Contributes Almost Nothing: The agent scaffolding architecture explained just 1.5% of the variance.
- Capability Ceilings: Stronger frontier models generate more hypothesis and evidence nodes per trace, but the topological structure of their reasoning remains fundamentally identical to weaker baselines. They retrieve better facts and execute code more cleanly, but they do not reason with greater scientific discipline.
In short: you cannot prompt, route, or orchestrate your way around a foundation model that lacks native epistemic verification.
The Epistemic Graveyard: Mapping Reasoning Graph Breakdowns
To audit agent logic at scale, the Corral team built custom environments connected to real scientific infrastructure—including live Atomic Force Microscopy (AFM), LAMMPS molecular dynamics simulations, qualitative inorganic wet-lab color reaction simulators, NMR spectra databases, and retrosynthetic reaction rule graphs.
Every agent trace was decomposed into an Epistemological Graph, labeling discrete operations: Hypotheses (H), Experiments/Tools (T), Evidence gathering (E), Refutations (R), and Belief Revisions (B). Mapping these graphs across 25,000+ executions exposed widespread anti-patterns:
- Evidence Non-Uptake (68%): In over two-thirds of all runs, an agent successfully invoked a tool, received unambiguous quantitative or qualitative feedback, and completely disregarded that feedback in its subsequent reasoning step.
- Untested Claims (53% to 63%): Agents regularly stated strong factual assertions (e.g., claiming a specific functional group or molecular symmetry) without ever issuing an experimental or computational check to verify the claim. In purely hypothesis-driven domains, untested claims spiked to 63%.
- Belief Stagnation: In a majority of traces, an agent's initial hypothesis remained static from start to finish. When confronted with direct experimental contradictions, refutation-driven belief revision occurred in only 26% of cases.
- Absence of Multi-Test Convergence (7%): In human science, robust discoveries rely on convergent validity—multiple independent tests pointing to the same conclusion. In Corral, convergent multi-test evidence appeared in just 7% of traces.
Epistemic Metric Rate Across 25k Runs
─────────────────────────────────────────────────────────
Evidence Ignored (Non-Uptake) 68%
Beliefs Never Updated Majority of runs
Untested Claims Asserted 53% (63% in Hypothesis Tasks)
Successful Refutation-Revision 26%
Convergent Multi-Test Evidence 7%
Invariance to Epistemic Demand
A human chemist or physicist fundamentally alters their cognitive strategy depending on the nature of the problem. Qualitative ion identification demands tight hypothesis-falsification cycles; retrosynthesis requires combinatorial tree search; molecular dynamics calibration requires simulate-validate-refine optimization.
Corral revealed that LLM agents exhibit epistemic invariance: they deploy the exact same reasoning topology regardless of the task type or problem scope.
High rates of evidence non-uptake appeared across both routine workflows and hypothesis-driven domains. Even when researchers increased problem complexity within a domain, agents did not scale their verification or falsification steps. They treated every challenge as a linear token-completion problem rather than an exploratory search under uncertainty.
The Decay of Reliability: Why Pass^k Collapses
When benchmarks only score final answers (e.g., whether the agent guessed the right molecule name), lucky hallucinations get scored as successes. But when you test whether an agent can repeat that success across repeated trials—measured as Pass^k consistency—the illusion evaporates.
In hypothesis-driven domains, unreliability compounded across repeated trials in epistemically demanding domains.
1.0 ──┐
│ Pass^1 (Initial Lucky Success)
0.5 └───┐
└───┐ Pass^2
0.0 └─────────────> Pass^5 ≈ 0.0 (Reliability Collapse)
To test whether agents could recover if provided high-quality guidance, the team ran Trace Interventions, injecting near-complete, human-verified successful reasoning trajectories directly into the agent's context window. Even when handed near-complete successful reasoning trajectories, models frequently failed to complete the remaining inferential step, falling back to baseless guesses instead of executing the logical conclusion demanded by the context.
Why Outcome-Based AI Benchmarks Must Die
The central conclusion of the Corral study extends well beyond chemistry and materials science: outcome-based evaluation is blind to epistemic failure.
If an autonomous agent is evaluating a drug candidate, auditing code for security vulnerabilities, or adjudicating financial risk, a correct final output derived from an invalid, un-falsified reasoning trace is dangerous. It cannot generalize to out-of-distribution conditions because the underlying process is ungrounded.
For the AI research community, Corral points toward three critical shifts:
- Stop Over-Investing in Complex Scaffolds: Prompt wrappers and agentic frameworks provide diminishing returns. If the foundational model cannot track logical falsification, adding layers of orchestration only inflates context tokens and latency.
- Make Epistemic Graphs Part of Evaluation: Benchmarks must audit reasoning graphs for evidence uptake, hypothesis testing, and belief revision rather than binary task completion flags.
- Train for Popperian Falsification: Current reinforcement learning from verifiable rewards (RLVR) and CoT distillation optimize for reaching target strings. Frontier labs must begin optimizing base models directly against epistemic criteria—penalizing models that ignore gathered evidence or fail to update beliefs in the presence of refutation.
Until foundational models are explicitly trained on the disciplined process of inquiry, AI "scientists" will remain sophisticated workflow automatons—capable of running tools, but incapable of practicing science.
Sources
The stories that matter for people who build with AI, with the source for every claim. Free, no spam.
Feed, daily deep-dive and bytes — readable offline, with push alerts for the topics you follow.