# Embodied AI Agents: Modeling the World

arXiv: https://arxiv.org/abs/2506.22355 · HTML: https://mlsanity.com/paper/2506.22355/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in World models for navigation, one of 11 directions in World models.
- Builds on 92 of its 133 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Placed in

- World models / [World models for navigation](https://mlsanity.com/wm/world-models-for-navigation/)

## Builds on (most cited first, 30 of 92)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)

## Built on by (most cited first, 30 of 61)

- [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://mlsanity.com/paper/2508.09736/) (2025)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players](https://mlsanity.com/paper/2605.28816/) (2026)
- [Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight](https://mlsanity.com/paper/2510.08713/) (2025)
- [Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test](https://mlsanity.com/paper/2601.04137/) (2026)
- [Action100M: A Large-scale Video Action Dataset](https://mlsanity.com/paper/2601.10592/) (2026)
- [VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model](https://mlsanity.com/paper/2605.01194/) (2026)
- [VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction](https://mlsanity.com/paper/2602.13294/) (2026)
- [Graph World Models: Concepts, Taxonomy, and Future Directions](https://mlsanity.com/paper/2604.27895/) (2026)
- [Embodied AI: Emerging Risks and Opportunities for Policy Action](https://mlsanity.com/paper/2509.00117/) (2025)
- [MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation](https://mlsanity.com/paper/2509.21797/) (2025)
- [Dynamic Reflections: Probing Video Representations with Text Alignment](https://mlsanity.com/paper/2511.02767/) (2025)
- [MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents](https://mlsanity.com/paper/2511.23055/) (2025)
- [DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling](https://mlsanity.com/paper/2512.03000/) (2025)
- [Aligning Agentic World Models via Knowledgeable Experience Learning](https://mlsanity.com/paper/2601.13247/) (2026)
- [From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents](https://mlsanity.com/paper/2602.04326/) (2026)
- [GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking](https://mlsanity.com/paper/2602.17555/) (2026)
- [SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control](https://mlsanity.com/paper/2605.22894/) (2026)
- [PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agents](https://mlsanity.com/paper/2509.19843/) (2025)
- [Memory Transfer Planning: LLM-driven Context-Aware Code Adaptation for Robot Manipulation](https://mlsanity.com/paper/2509.24160/) (2025)
- [ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replay](https://mlsanity.com/paper/2511.19033/) (2025)
- [Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints](https://mlsanity.com/paper/2603.11755/) (2026)
- [StreamingClaw Technical Report](https://mlsanity.com/paper/2603.22120/) (2026)
- [Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion](https://mlsanity.com/paper/2604.08780/) (2026)
- [Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recovery](https://mlsanity.com/paper/2605.01191/) (2026)

## Abstract (arXiv)

This paper describes our research on AI agents embodied in visual, virtual or physical forms, enabling them to interact with both users and their environments. These agents, which include virtual avatars, wearable devices, and robots, are designed to perceive, learn and act within their surroundings, which makes them more similar to how humans learn and interact with the environments as compared to disembodied agents. We propose that the development of world models is central to reasoning and planning of embodied AI agents, allowing these agents to understand and predict their environment, to understand user intentions and social contexts, thereby enhancing their ability to perform complex tasks autonomously. World modeling encompasses the integration of multimodal perception, planning through reasoning for action and control, and memory to create a comprehensive understanding of the physical world. Beyond the physical world, we also propose to learn the mental world model of users to enable better human-agent collaboration.