mlsanityData to 7 Oct 2026

Can AI Scientists Change Their Minds? Prior-Evidence Conflict in Synthetic Universes

arXiv 2609.36726 ↗First posted 29 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Can a scientific agent distinguish a law it inferred from evidence from one it merely recognizes? We introduce Synthetic Universes, a controlled benchmark that pairs canonical famous worlds with matched twisted twins governed by nearby noncanonical mechanisms. We evaluate each reported law twice: by executing it on held-out continuations and transfer settings, and by independently checking whether it recovers the generating mechanism. In the current checkpoint of a pre-specified 60-cell study, 22 trials were graded and one additional run ended in infrastructure failure. Among 20 twin trials, 8 pass predictive verification while 5 recover the generator. The dissociation is bidirectional: six parsable outputs predict successfully while missing the mechanism, whereas three recover the mechanism but fail predictive rollout. Drag exhibits the first pattern (5/5 predictive pass, 1/5 mechanism recovery); Gravity exhibits the second (1/5 predictive pass, 4/5 mechanism recovery). Because matched famous controls, the corrected identifiability sweep, and the Evidence Ladder remain incomplete, we do not claim a confirmatory causal prior-conflict effect. Instead, the completed runs establish a narrower verification result: predictive adequacy and mechanism recovery are distinct scientific claims and require distinct tests.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

25 of 44 references.

References · 25

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  4. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  5. 2019AI Feynman: a Physics-Inspired Method for Symbolic Regressioncited by 536
  6. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  7. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  8. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  9. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  10. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  11. 2026Towards End-to-End Automation of AI Researchcited by 150
  12. 2020AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularitycited by 140
13 more
  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  2. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 117
  3. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
  4. 2025LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Modelscited by 61
  5. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 57
  6. 2025ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decompositioncited by 57
  7. 2025Evaluating Large Language Models in Scientific Discoverycited by 38
  8. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 32
  9. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarioscited by 27
  10. 2026Accelerating Scientific Research with Gemini: Case Studies and Common Techniquescited by 24
  11. 2026The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Researchcited by 6
  12. 2026Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradictscited by 3
  13. 2025LLMs, Reasoning and Plagiarismcited by 2

Built on by

Not cited yet.