mlsanityData to 7 Oct 2026

V-IRL: Grounding Virtual Intelligence in Real Life

arXiv 2402.03310 ↗First posted 5 Feb 2024cs.AI cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

There is a sensory gulf between the Earth that humans inhabit and the digital realms in which modern AI agents are created. To develop AI agents that can sense, think, and act as flexibly as humans in real-world settings, it is imperative to bridge the realism gap between the digital and physical worlds. How can we embody agents in an environment as rich and diverse as the one we inhabit, without the constraints imposed by real hardware and control? Towards this end, we introduce V-IRL: a platform that enables agents to scalably interact with the real world in a virtual yet realistic environment. Our platform serves as a playground for developing agents that can accomplish various practical tasks and as a vast testbed for measuring progress in capabilities spanning perception, decision-making, and interaction with real-world data across the entire globe.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 73 of its 92 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

73 of 92 references.

References · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  13. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  14. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  15. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  16. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  17. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  18. 2023PaLM 2 Technical Reportcited by 1,154
  19. 2021Grounded Language-Image Pre-trainingcited by 1,147
  20. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  21. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  24. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  25. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  26. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  27. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  28. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  29. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  30. 2022Language-driven Semantic Segmentationcited by 711
  31. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  32. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  33. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  34. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  35. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  36. 2018On Evaluation of Embodied Navigation Agentscited by 526
  37. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  38. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 8; 2026: 13.

35 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  3. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  4. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  5. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  6. 2024EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environmentcited by 42
  7. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  8. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  9. 2024CityBench: Evaluating the Capabilities of Large Language Models for Urban Taskscited by 26
  10. 2024When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environmentscited by 25
  11. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  12. 2024UrbanWorld: An Urban World Model for 3D City Generationcited by 23
23 more
  1. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  2. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  3. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
  4. 2025UrbanFeel: A Comprehensive Benchmark for Temporal and Perceptual Understanding of City Scenes through Human Perspectivecited by 7
  5. 2025Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Framescited by 6
  6. 2024TravelAgent: Generative Agents in the Built Environmentcited by 5
  7. 2025H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understandingcited by 2
  8. 2026Cambrian-P: Pose-Grounded Video Understandingcited by 2
  9. 2025VC-Agent: An Interactive Agent for Customized Video Dataset Collectioncited by 1
  10. 2026Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Visioncited by 1
  11. 2026Less Detail, Better Answers: Degradation-Driven Prompting for VQAcited by 1
  12. 2026Visually-grounded Humanoid Agentscited by 1
  13. 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMscited by 1
  14. 2026Do VLMs See What Sensors Feel? A Scalable Expert-Guided Design for Wheelchair Accessibility Assessment from Street Viewcited by 1
  15. 2026360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agentscited by 1
  16. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  17. 2026DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculumcited by 1
  18. 2025TraveLLaMA: A Multimodal Travel Assistant with Large-Scale Dataset and Structured Reasoningcited by 0
  19. 2025CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understandingcited by 0
  20. 2026From Pixels to Digital Agents: An Empirical Study on the Taxonomy and Technological Trends of Reinforcement Learning Environmentscited by 0
  21. 2026Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learningcited by 0
  22. 2026MAP: A Benchmark on Multimodal Accessibility Planning for Real World Placescited by 0
  23. 2026A Vision-Language Framework for Measuring Social Life on Sidewalkscited by 0