# VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications

arXiv: https://arxiv.org/abs/2405.11537 · HTML: https://mlsanity.com/paper/2405.11537/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 19 May 2024.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents](https://mlsanity.com/paper/2401.12963/) (2024)
- [GraspLook: a VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment](https://mlsanity.com/paper/2110.12518/) (2021)
- [CognitiveDog: Large Multimodal Model Based System to Translate Vision and Language into Action of Quadruped Robot](https://mlsanity.com/paper/2401.09388/) (2024)
- [CognitiveOS: Large Multimodal Model based System to Endow Any Type of Robot with Generative AI](https://mlsanity.com/paper/2401.16205/) (2024)

## Built on by (most cited first, 9 of 9)

- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models](https://mlsanity.com/paper/2509.17664/) (2025)
- [FaceXBench: Evaluating Multimodal LLMs on Face Understanding](https://mlsanity.com/paper/2501.10360/) (2025)
- [SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards](https://mlsanity.com/paper/2511.07403/) (2025)
- [Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models](https://mlsanity.com/paper/2507.16524/) (2025)
- [Refer to Any Segmentation Mask Group With Vision-Language Prompts](https://mlsanity.com/paper/2506.05342/) (2025)
- [Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes](https://mlsanity.com/paper/2410.22177/) (2024)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [Towards user-centered interactive medical image segmentation in VR with an assistive AI agent](https://mlsanity.com/paper/2505.07214/) (2025)

## Abstract (arXiv)

The advent of immersive Virtual Reality applications has transformed various domains, yet their integration with advanced artificial intelligence technologies like Visual Language Models remains underexplored. This study introduces a pioneering approach utilizing VLMs within VR environments to enhance user interaction and task efficiency. Leveraging the Unity engine and a custom-developed VLM, our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions. The incorporation of speech-to-text and text-to-speech technologies allows for seamless communication between the user and the VLM, enabling the system to guide users through complex tasks effectively. Preliminary experimental results indicate that utilizing VLMs not only reduces task completion times but also improves user comfort and task engagement compared to traditional VR interaction methods.