# Virtual Community: An Open World for Humans, Robots, and Society

arXiv: https://arxiv.org/abs/2508.14893 · HTML: https://mlsanity.com/paper/2508.14893/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 91 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Builds on (most cited first, 30 of 75)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)

## Built on by (most cited first, 13 of 13)

- [SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds](https://mlsanity.com/paper/2512.01078/) (2025)
- [Articulation in Motion: Prior-free Part Mobility Analysis for Articulated Objects By Dynamic-Static Disentanglement](https://mlsanity.com/paper/2603.02910/) (2026)
- [TongSIM: A General Platform for Simulating Intelligent Machines](https://mlsanity.com/paper/2512.20206/) (2025)
- [UrbanWorld2.0: A Multimodal Agentic Framework for Reality-Aligned 3D World Generation at City-Scale](https://mlsanity.com/paper/2511.18005/) (2025)
- [AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents](https://mlsanity.com/paper/2606.24893/) (2026)
- [Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments](https://mlsanity.com/paper/2509.17425/) (2025)
- [FreeAskWorld: An Interactive and Closed-Loop Simulator for Human-Centric Embodied AI](https://mlsanity.com/paper/2511.13524/) (2025)
- [UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City](https://mlsanity.com/paper/2608.27456/) (2026)
- [S3IT: A Benchmark for Spatially Situated Social Intelligence Test](https://mlsanity.com/paper/2512.19992/) (2025)
- [Sentinel: Embodied Cooperative Spatial Reasoning and Planning](https://mlsanity.com/paper/2605.26239/) (2026)
- [NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation](https://mlsanity.com/paper/2607.19695/) (2026)
- [SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnership](https://mlsanity.com/paper/2609.19610/) (2026)
- [Generative Embodied Multiple Behavior Control Systems for Human-like Agents](https://mlsanity.com/paper/2609.22691/) (2026)

## Abstract (arXiv)

The rapid progress in AI and Robotics may lead to a profound societal transformation, as humans and robots begin to coexist within shared communities, introducing both opportunities and challenges. To explore this future, we present Virtual Community-an open-world platform for humans, robots, and society-built on a universal physics engine and grounded in real-world 3D scenes. With Virtual Community, we aim to enable the study of embodied social intelligence at scale. To support these, Virtual Community features: 1) An open-source multi-agent physics simulator that supports robots, humans, and their interactions within a society; 2) A large-scale, real-world aligned community generation pipeline, including vast outdoor space, diverse indoor scenes, and a community of grounded agents with rich characters and appearances. Leveraging Virtual Community, we propose two novel challenges. The Community Planning Challenge evaluates multi-agent reasoning and planning ability in open-world settings, such as cooperating to help agents with daily activities and efficiently connecting other agents. The Community Robot Challenge requires multiple heterogeneous robots to collaborate in solving complex open-world tasks. We evaluate various baselines on these tasks and demonstrate the challenges in both high-level open-world task planning and low-level cooperation controls. We hope that Virtual Community will unlock further study of human-robot coexistence within open-world environments.