mlsanityData to 13 Oct 2026

CausalVerify: End-to-End Verification of Causal Analyses by Language Models

arXiv 2609.07944 ↗First posted 7 Sep 2026cs.AI cs.CL econ.EMcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models increasingly perform empirical analyses end to end, yet existing evaluations assess the written explanation or whether generated code executes, not whether the executed workflow recovers the intended causal estimand. We introduce CausalVerify, an execution-grounded benchmark for end-to-end causal analysis that follows a model from research-context interpretation to estimand recovery. It scores this workflow at four distinct layers: method recognition, design specification, executable implementation, and estimand recovery. It combines 259 real-paper contexts, 100 fixed-seed synthetic scenarios with executable reference estimates, and 23 paper-twin pairs in which a model commits to a design before seeing the data and its executed analysis is scored against a canonical estimator on the same realised dataset. Execution is not correctness. Among 426 model-written workflows that run without error, 15.5% fail verification, and a keyword score of the effect direction stated in the text is a poor proxy for recovery. In a 23-pair, nine-model paired study, replacing a model's committed design with the reference design and its execution conventions raises joint recovery of the point estimate and standard error from 15.0% to 51.5%; yet 48.5% of eligible seeds still fail under the reference design. The direction replicates on six pairs built afterwards under a frozen construction protocol, although on the two newest pairs the gain is confined to models from the family that built the references. Design specification is consequential but not sufficient: a plausible method and runnable code do not guarantee recovery, and even supplying the reference design and its conventions leaves substantial downstream failure. CausalVerify evaluates the executed workflow rather than its surface plausibility, and every reported number is recomputed from frozen artifacts by a single script.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

34 of 57 references.

References · 34

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,362
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 960
  7. 2021Measuring Coding Challenge Competence With APPScited by 891
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 800
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 740
  10. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 533
  11. 2021Dynabench: Rethinking Benchmarking in NLPcited by 452
  12. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 447
22 more
  1. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 353
  2. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 352
  3. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 344
  4. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 226
  5. 2026Towards End-to-End Automation of AI Researchcited by 152
  6. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 150
  7. 2023Can Large Language Models Infer Causation from Correlation?cited by 143
  8. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 106
  9. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 105
  10. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?cited by 37
  11. 2024Causal Inference with Large Language Model: A Surveycited by 31
  12. 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMscited by 20
  13. 2022What's Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literaturecited by 18
  14. 2024CLEAR: Can Language Models Really Understand Causal Graphs?cited by 14
  15. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Modelscited by 14
  16. 2026ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciencescited by 9
  17. 2026CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimationcited by 6
  18. 2026Toward Full Autonomous Laboratory Instrumentation Control with Large Language Modelscited by 5
  19. 2026Read the Paper, Write the Code: Agentic Reproduction of Social-Science Resultscited by 5
  20. 2026InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systemscited by 3
  21. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Meancited by 3
  22. 2026CausalDS: Benchmarking Causal Reasoning in Data-Science Agentscited by 1

Built on by

Not cited yet.