# Higher-Order Belief in Incomplete Information MAIDs

arXiv: https://arxiv.org/abs/2503.06323 · HTML: https://mlsanity.com/paper/2503.06323/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 38 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 9 of 9)

- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [The Reasons that Agents Act: Intention and Instrumental Goals](https://mlsanity.com/paper/2402.07221/) (2024)
- [Path-Specific Objectives for Safer Agent Incentives](https://mlsanity.com/paper/2204.10018/) (2022)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [Equilibrium Refinements for Multi-Agent Influence Diagrams: Theory and Practice](https://mlsanity.com/paper/2102.05008/) (2021)
- [Networks of Influence Diagrams: A Formalism for Representing Agents' Beliefs and Decision-Making Processes](https://mlsanity.com/paper/1401.3426/) (2014)
- [Incentives for Responsiveness, Instrumental Control and Impact](https://mlsanity.com/paper/2001.07118/) (2020)

## Built on by (most cited first, 1 of 1)

- [The Impossibility of Eliciting Latent Knowledge](https://mlsanity.com/paper/2606.12268/) (2026)

## Abstract (arXiv)

Multi-agent influence diagrams (MAIDs) are probabilistic graphical models which represent strategic interactions between agents. MAIDs are equivalent to extensive form games (EFGs) but have a more compact and informative structure. However, MAIDs cannot, in general, represent settings of incomplete information -- wherein agents have different beliefs about the game being played, and different beliefs about each-other's beliefs. In this paper, we introduce incomplete information MAIDs (II-MAIDs). We define both infinite and finite-depth II-MAIDs and prove an equivalence relation to EFGs with incomplete information and no common prior over types. We prove that II-MAIDs inherit classical equilibria concepts via this equivalence, but note that these solution concepts are often unrealistic in the setting with no common prior because they violate common knowledge of rationality. We define a more realistic solution concept based on recursive best-response. Throughout, we describe an example with a hypothetical AI agent undergoing evaluation to illustrate the applicability of II-MAIDs.