TongSIM: A General Platform for Simulating Intelligent Machines
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
As artificial intelligence (AI) rapidly advances, especially in multimodal large language models (MLLMs), research focus is shifting from single-modality text processing to the more complex domains of multimodal and embodied AI. Embodied intelligence focuses on training agents within realistic simulated environments, leveraging physical interaction and action feedback rather than conventionally labeled datasets. Yet, most existing simulation platforms remain narrowly designed, each tailored to specific tasks. A versatile, general-purpose training environment that can support everything from low-level embodied navigation to high-level composite activities, such as multi-agent social simulation and human-AI collaboration, remains largely unavailable. To bridge this gap, we introduce TongSIM, a high-fidelity, general-purpose platform for training and evaluating embodied agents. TongSIM offers practical advantages by providing over 100 diverse, multi-room indoor scenarios as well as an open-ended, interaction-rich outdoor town simulation, ensuring broad applicability across research needs. Its comprehensive evaluation framework and benchmarks enable precise assessment of agent capabilities, such as perception, cognition, decision-making, human-robot cooperation, and spatial and social reasoning. With features like customized scenes, task-adaptive fidelity, diverse agent types, and dynamic environmental simulation, TongSIM delivers flexibility and scalability for researchers, serving as a unified platform that accelerates training, evaluation, and advancement toward general embodied intelligence.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 63 references that are in the map. First posted on arXiv 23 Dec 2025.
Builds on
49 of 63 references.
Closest ancestors · 1
Same direction, newest first
Other references · 48
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2015VQA: Visual Question Answeringcited by 2,489
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
36 more
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
- 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
- 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
- 2018VirtualHome: Simulating Household Activities via Programscited by 411
- 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
- 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
- 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
- 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
- 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
- 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
- 2022Masked World Models for Visual Controlcited by 169
- 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
- 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
- 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89
- 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
- 2025LLM Agents for Education: Advances and Applicationscited by 59
- 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
- 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 42
- 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
- 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
- 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
- 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
- 2024CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMscited by 17
- 2023Where Are We So Far? Understanding Data Storytelling Tools from the Perspective of Human-AI Collaborationcited by 13
- 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
- 2023Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastinationcited by 12
- 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 10
- 2025CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarioscited by 10
- 2019VRGym: A Virtual Testbed for Physical and Interactive AIcited by 9
- 2024Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraftcited by 9
- 2024On the Emergence of Symmetrical Realitycited by 8
- 2025DVM: Towards Controllable LLM Agents in Social Deduction Gamescited by 6
- 2019Symmetrical Reality: Toward a Unified Framework for Physical and Virtual Realitycited by 4
- 2018Inverse Augmented Reality: A Virtual Agent's Perspectivecited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
- 2025Embodied AI: From LLMs to World Models
- 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
- 2026VirtualEnv: A Platform for Embodied AI Research
- 2024LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments
- 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
- 2024EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment
- 2026Lingjing: A Simulation Testbed for Multi-Agent Embodied Tasks in Open-Ended Cities