# VRGym: A Virtual Testbed for Physical and Interactive AI

arXiv: https://arxiv.org/abs/1904.01698 · HTML: https://mlsanity.com/paper/1904.01698/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 9 of 9)

- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance](https://mlsanity.com/paper/2103.14231/) (2021)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)
- [On the Learning Mechanisms in Physical Reasoning](https://mlsanity.com/paper/2210.02075/) (2022)
- [On the Emergence of Symmetrical Reality](https://mlsanity.com/paper/2401.15132/) (2024)
- [Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene](https://mlsanity.com/paper/2108.02846/) (2021)
- [TongSIM: A General Platform for Simulating Intelligent Machines](https://mlsanity.com/paper/2512.20206/) (2025)
- [Body Discovery of Embodied AI](https://mlsanity.com/paper/2503.19941/) (2025)

## Abstract (arXiv)

We propose VRGym, a virtual reality testbed for realistic human-robot interaction. Different from existing toolkits and virtual reality environments, the VRGym emphasizes on building and training both physical and interactive agents for robotics, machine learning, and cognitive science. VRGym leverages mechanisms that can generate diverse 3D scenes with high realism through physics-based simulation. We demonstrate that VRGym is able to (i) collect human interactions and fine manipulations, (ii) accommodate various robots with a ROS bridge, (iii) support experiments for human-robot interaction, and (iv) provide toolkits for training the state-of-the-art machine learning algorithms. We hope VRGym can help to advance general-purpose robotics and machine learning agents, as well as assisting human studies in the field of cognitive science.