mlsanityData to 14 Oct 2026

The Limits of Predicting Agents from Behaviour

arXiv 2506.02923 ↗First posted 3 Jun 2025cs.AI stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as if it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can we infer an agent's beliefs from their behaviour, and how reliably can these inferred beliefs predict the agent's behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent's behaviour is guided by a world model. Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments, which represent a theoretical limit for predicting intentional agents from behavioural data alone. We discuss the implications of these results for several research areas including fairness and safety.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 63 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

28 of 63 references.

References · 28

  1. 2017Counterfactual Fairnesscited by 804
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  3. 2021Causal Abstractions of Neural Networkscited by 422
  4. 2023Language Models Represent Space and Timecited by 316
  5. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  7. 2022Talking About Large Language Modelscited by 175
  8. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  9. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  10. 2013Probabilities of Causation: Bounds and Identificationcited by 90
  11. 2024Robust agents learn causal world modelscited by 81
  12. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 75
16 more
  1. 2021Chess as a Testbed for Language Model State Trackingcited by 72
  2. 2021Partial Counterfactual Identification from Observational and Experimental Datacited by 64
  3. 2021Agent Incentives: A Causal Perspectivecited by 51
  4. 2021Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confoundingcited by 50
  5. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  6. 2018Bounds on the conditional and average treatment effect with unobserved confounding factorscited by 39
  7. 2022Personalized Decision Making -- A Conceptual Introductioncited by 30
  8. 2022Counterfactual harmcited by 29
  9. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  10. 2016Towards Resolving Unidentifiability in Inverse Reinforcement Learningcited by 18
  11. 2022A Causal Analysis of Harmcited by 15
  12. 2024Does ChatGPT Have a Mind?cited by 14
  13. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 13
  14. 2020Deriving Bounds and Inequality Constraints Using LogicalRelations Among Counterfactualscited by 8
  15. 2023Towards Bounding Causal Effects under Markov Equivalencecited by 8
  16. 2025Partial Transportability for Domain Generalizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agentscited by 1
  2. 2026The Impossibility of Eliciting Latent Knowledgecited by 1
  3. 2026When Are Aggregate Agent Traces Diagnosable? Traffic-Governed Interpretation and Calibrated Abstentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavioral Inference at Scale: The Fundamental Asymmetry Between Motivations and Belief Systems
  2. 2024Limitations of Agents Simulated by Predictive Models
  3. 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
  4. 2025General agents contain world models
  5. 2025AI Agent Behavioral Science
  6. 2026Position: Behavioral Systems Require Behavioral Tests
  7. 2025Agentic AI Needs a Systems Theory
  8. 2026Radical AI Interpretability