# Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs

arXiv: https://arxiv.org/abs/2004.12248 · HTML: https://mlsanity.com/paper/2004.12248/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2020.

## Built on by (most cited first, 8 of 8)

- [YouRefIt: Embodied Reference Understanding with Language and Gesture](https://mlsanity.com/paper/2109.03413/) (2021)
- [Learning Triadic Belief Dynamics in Nonverbal Communication from Videos](https://mlsanity.com/paper/2104.02841/) (2021)
- [Sequential Manipulation Planning on Scene Graph](https://mlsanity.com/paper/2207.04364/) (2022)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)
- [A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics](https://mlsanity.com/paper/2103.01403/) (2021)
- [MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generation](https://mlsanity.com/paper/2306.15253/) (2023)
- [Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain](https://mlsanity.com/paper/2507.02016/) (2025)
- [IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning](https://mlsanity.com/paper/2608.23330/) (2026)

## Abstract (arXiv)

Aiming to understand how human (false-)belief--a core socio-cognitive ability--would affect human interactions with robots, this paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs. Specifically, a parse graph (pg) is learned from a single-view spatiotemporal parsing by aggregating various object states along the time; such a learned representation is accumulated as the robot's knowledge. An inference algorithm is derived to fuse individual pg from all robots across multi-views into a joint pg, which affords more effective reasoning and inference capability to overcome the errors originated from a single view. In the experiments, through the joint inference over pg-s, the system correctly recognizes human (false-)belief in various settings and achieves better cross-view accuracy on a challenging small object tracking dataset.