mlsanityData to 14 Oct 2026

ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence

arXiv 2610.14381 ↗First posted 9 Oct 2026cs.AIcited by 0

On the map: LLM agents and tool use → Medical and scientific domain agents also fits: Web, search, and research agents

Abstract (arXiv)

Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt. A useful evaluation must let agents choose experiments, interpret finite evidence, and make causal decisions, while keeping the environment coherent across repeated observations and interventions and preserving objective ground truth. We introduce ACED-BENCH (Active Causal Evidence and Decision Benchmark), a neuro-symbolic benchmark built around this requirement. The key idea is to separate the readable scientific surface from the formal causal world: LLMs generate natural-language scenarios and task text, while hidden probabilistic graphical models define the causal graph, data-generating process, intervention semantics, validation checks, and oracle answers. This yields scientific worlds that are natural for agents to investigate but exact enough for scalable, programmatic evaluation. The main leaderboard ACED-CORE contains 180 tasks: 60 set-valued causal ancestry/descendancy questions and 120 constrained causal-decision tasks. We also report ACED-QUAL, a 150-question yes/no qualification check using the same hidden-PGM interface. We evaluate seven contemporary LLMs and two earlier-generation models across three active agent designs. ACED-QUAL checks whether agents can use the interface to draw conclusions from evidence, while ACED-CORE tests whether they can decide what evidence to collect and when to commit to an answer. Evidence-ledger analysis localizes failures within the investigation: agents test the wrong intervention, lose relevant evidence, or continue after sufficient evidence has appeared. These results suggest that reliable AI co-scientists need explicit support for experiment design, evidence retention, answer commitment, and stopping.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 32 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

24 of 32 references.

References · 24

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  5. 2023Can Large Language Models Infer Causation from Correlation?cited by 146
  6. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  7. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 109
  8. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 107
  9. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  10. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 77
  11. 2023Causal Discovery with Language Models as Imperfect Expertscited by 64
  12. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 63
12 more
  1. 2022Active Bayesian Causal Inferencecited by 35
  2. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 33
  3. 2024Integrating Large Language Models in Causal Discovery: A Statistical Causal Approachcited by 32
  4. 2022Active Learning for Optimal Intervention Design in Causal Modelscited by 31
  5. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agentscited by 31
  6. 2024Are Large Language Models Good Statisticians?cited by 27
  7. 2024What Are the Odds? Language Models Are Capable of Probabilistic Reasoningcited by 26
  8. 2025Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agentscited by 24
  9. 2025Can Large Language Models Help Experimental Design for Causal Discovery?cited by 11
  10. 2025Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inferencecited by 6
  11. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 6
  12. 2025OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World Datacited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
  2. 2026CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
  3. 2025Causal MAS: A Survey of Large Language Model Architectures for Discovery and Effect Estimation
  4. 2026Why LLMs Fail at Causal Discovery and How Interventional Agents Escape
  5. 2024CausalGraph2LLM: Evaluating LLMs for Causal Queries
  6. 2026CausalArena: Benchmarking Causal Discovery in the Foundation Model Era
  7. 2026FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
  8. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models