# Text2VR: Automated instruction Generation in Virtual Reality using Large language Models for Assembly Task

arXiv: https://arxiv.org/abs/2508.03699 · HTML: https://mlsanity.com/paper/2508.03699/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 37 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation](https://mlsanity.com/paper/2308.01240/) (2023)
- [LLMR: Real-time Prompting of Interactive Worlds using Large Language Models](https://mlsanity.com/paper/2309.12276/) (2023)
- [Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning](https://mlsanity.com/paper/2408.14774/) (2024)
- [Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models](https://mlsanity.com/paper/2310.13127/) (2023)
- [A Virtual Reality Training System for Automotive Engines Assembly and Disassembly](https://mlsanity.com/paper/2311.02108/) (2023)
- [CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR](https://mlsanity.com/paper/2411.04671/) (2024)
- [Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation](https://mlsanity.com/paper/2502.02201/) (2025)
- [Virtual Reality in Manufacturing Education: A Scoping Review Indicating State-of-the-Art, Benefits, and Challenges Across Domains, Levels, and Entities](https://mlsanity.com/paper/2503.18805/) (2025)

## Abstract (arXiv)

Virtual Reality (VR) has emerged as a powerful tool for workforce training, offering immersive, interactive, and risk-free environments that enhance skill acquisition, decision-making, and confidence. Despite its advantages, developing VR applications for training remains a significant challenge due to the time, expertise, and resources required to create accurate and engaging instructional content. To address these limitations, this paper proposes a novel approach that leverages Large Language Models (LLMs) to automate the generation of virtual instructions from textual input. The system comprises two core components: an LLM module that extracts task-relevant information from the text, and an intelligent module that transforms this information into animated demonstrations and visual cues within a VR environment. The intelligent module receives input from the LLM module and interprets the extracted information. Based on this, an instruction generator creates training content using relevant data from a database. The instruction generator generates the instruction by changing the color of virtual objects and creating animations to illustrate tasks. This approach enhances training effectiveness and reduces development overhead, making VR-based training more scalable and adaptable to evolving industrial needs.