# Modeling Others' Minds as Code arXiv: https://arxiv.org/abs/2510.01272 ยท HTML: https://mlsanity.com/paper/2510.01272/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 52 of its 101 references that are in the map. - First posted on arXiv 29 Sep 2025. ## Builds on (most cited first, 30 of 52) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018) - [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019) - [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019) - [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025) - [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018) - [Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning](https://mlsanity.com/paper/2305.14909/) (2023) - [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023) - [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020) - [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015) - [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024) - [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023) - [WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment](https://mlsanity.com/paper/2402.12275/) (2024) - [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023) - [Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities](https://mlsanity.com/paper/2311.10227/) (2023) - [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024) - [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020) - [Learning to Synthesize Programs as Interpretable and Generalizable Policies](https://mlsanity.com/paper/2108.13643/) (2021) - [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020) - [Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent States](https://mlsanity.com/paper/2102.05261/) (2021) - [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021) - [Finding Friend and Foe in Multi-Agent Games](https://mlsanity.com/paper/1906.02330/) (2019) - [Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought](https://mlsanity.com/paper/2305.16744/) (2023) - [Learning adaptive planning representations with natural language guidance](https://mlsanity.com/paper/2312.08566/) (2023) - [Theory of Minds: Understanding Behavior in Groups Through Inverse Planning](https://mlsanity.com/paper/1901.06085/) (2019) - [Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Models](https://mlsanity.com/paper/2502.11881/) (2025) - [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024) - [AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling](https://mlsanity.com/paper/2502.15676/) (2025) - [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021) - [Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models](https://mlsanity.com/paper/2407.06004/) (2024) ## Built on by (most cited first, 7 of 7) - [A Probabilistic Framework for LLM-Based Model Discovery](https://mlsanity.com/paper/2602.18266/) (2026) - [Learning User Simulators with Turing Rewards](https://mlsanity.com/paper/2606.19336/) (2026) - [Evaluating LLMs in Open-Source Games](https://mlsanity.com/paper/2512.00371/) (2025) - [Grounding Social Perception in Intuitive Physics](https://mlsanity.com/paper/2603.27410/) (2026) - [SAATT Nav: a Socially Aware Autonomous Transparent Transportation Navigation Framework for Wheelchairs](https://mlsanity.com/paper/2603.13698/) (2026) - [RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments](https://mlsanity.com/paper/2606.26094/) (2026) - [Hypothesis-guided discovery of cognitive algorithms via program refinement](https://mlsanity.com/paper/2610.02523/) (2026) ## Abstract (arXiv) Accurate prediction of human behavior is essential for robust and safe human-AI collaboration. However, existing approaches for modeling people are often data-hungry and brittle because they either make unrealistic assumptions about rationality or are too computationally demanding to adapt rapidly. Our key insight is that many everyday social interactions may follow predictable patterns; efficient "scripts" that minimize cognitive load for actors and observers, e.g., "wait for the green light, then go." We propose modeling these routines as behavioral programs instantiated in computer code rather than policies conditioned on beliefs and desires. We introduce ROTE, a novel algorithm that leverages both large language models (LLMs) for synthesizing a hypothesis space of behavioral programs, and probabilistic inference for reasoning about uncertainty over that space. We test ROTE in a suite of gridworld tasks and a large-scale embodied household simulator. ROTE predicts human and AI behaviors from sparse observations, outperforming competitive baselines -- including behavior cloning and LLM-based methods -- by as much as 50% in terms of in-sample accuracy and out-of-sample generalization. By treating action understanding as a program synthesis problem, ROTE opens a path for AI systems to efficiently and effectively predict human behavior in the real-world.