# UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI

arXiv: https://arxiv.org/abs/2412.20977 · HTML: https://mlsanity.com/paper/2412.20977/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 70 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Builds on (most cited first, 30 of 47)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2312.03275/) (2023)
- [InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment](https://mlsanity.com/paper/2406.04882/) (2024)
- [Building Generalizable Agents with a Realistic and Rich 3D Environment](https://mlsanity.com/paper/1801.02209/) (2018)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [Offline Visual Representation Learning for Embodied Navigation](https://mlsanity.com/paper/2204.13226/) (2022)
- [Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning](https://mlsanity.com/paper/2212.08860/) (2022)

## Built on by (most cited first, 25 of 25)

- [UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning](https://mlsanity.com/paper/2505.15725/) (2025)
- [You Don't Need Pre-built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structures](https://mlsanity.com/paper/2508.06105/) (2025)
- [Virtual Community: An Open World for Humans, Robots, and Society](https://mlsanity.com/paper/2508.14893/) (2025)
- [SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds](https://mlsanity.com/paper/2512.01078/) (2025)
- [AirSim360: A Panoramic Simulation Platform within Drone View](https://mlsanity.com/paper/2512.02009/) (2025)
- [Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends](https://mlsanity.com/paper/2606.01164/) (2026)
- [ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment](https://mlsanity.com/paper/2503.02505/) (2025)
- [VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models](https://mlsanity.com/paper/2505.20718/) (2025)
- [DeliveryBench: Can Agents Earn Profit in Real World?](https://mlsanity.com/paper/2512.19234/) (2025)
- [EmbRACE-3K: Embodied Reasoning and Action in Complex Environments](https://mlsanity.com/paper/2507.10548/) (2025)
- [UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models](https://mlsanity.com/paper/2604.02241/) (2026)
- [3D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigation](https://mlsanity.com/paper/2605.26500/) (2026)
- [SPEAR: A Simulator for Photorealistic Embodied AI Research](https://mlsanity.com/paper/2607.06701/) (2026)
- [Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents](https://mlsanity.com/paper/2507.23698/) (2025)
- [LychSim: A Controllable and Interactive Simulation Framework for Vision Research](https://mlsanity.com/paper/2605.12449/) (2026)
- [CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agents](https://mlsanity.com/paper/2511.20216/) (2025)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations](https://mlsanity.com/paper/2608.15659/) (2026)
- [Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation](https://mlsanity.com/paper/2606.06836/) (2026)
- [CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking](https://mlsanity.com/paper/2607.15004/) (2026)
- [Bridging Values and Behavior: A Hierarchical Framework for Proactive Embodied Agents](https://mlsanity.com/paper/2604.27699/) (2026)
- [RescueBench: Can Embodied Agents Save Lives in the Wild ?](https://mlsanity.com/paper/2606.01848/) (2026)
- [PermaVid: Consistent Video Generation Across Edits via Disentangled Context Memory](https://mlsanity.com/paper/2606.16449/) (2026)
- [BinoGen: Scaling egocentric binocular data for embodied visual perception and learning](https://mlsanity.com/paper/2609.19881/) (2026)
- [WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents](https://mlsanity.com/paper/2609.40325/) (2026)

## Abstract (arXiv)

We introduce UnrealZoo, a collection of over 100 photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of open-world environments. We also provide a rich variety of playable entities, including humans, animals, robots, and vehicles for embodied AI research. We extend UnrealCV with optimized APIs and tools for data collection, environment augmentation, distributed training, and benchmarking. These improvements achieve significant improvements in the efficiency of rendering and communication, enabling advanced applications such as multi-agent interactions. Our experimental evaluation across visual navigation and tracking tasks reveals two key insights: 1) environmental diversity provides substantial benefits for developing generalizable reinforcement learning (RL) agents, and 2) current embodied agents face persistent challenges in open-world scenarios, including navigation in unstructured terrain, adaptation to unseen morphologies, and managing latency in the close-loop control systems for interacting in highly dynamic objects. UnrealZoo thus serves as both a comprehensive testing ground and a pathway toward developing more capable embodied AI systems for real-world deployment.