# From Voices to Worlds: Developing an AI-Powered Framework for 3D Object Generation in Augmented Reality

arXiv: https://arxiv.org/abs/2503.16474 · HTML: https://mlsanity.com/paper/2503.16474/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [WavLLM: Towards Robust and Adaptive Speech Large Language Model](https://mlsanity.com/paper/2404.00656/) (2024)
- [DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering](https://mlsanity.com/paper/2307.10173/) (2023)
- [Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender System](https://mlsanity.com/paper/2404.11343/) (2024)
- [MISAR: A Multimodal Instructional System with Augmented Reality](https://mlsanity.com/paper/2310.11699/) (2023)
- [LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking](https://mlsanity.com/paper/2404.05624/) (2024)
- [Toward Scalable and Controllable AR Experimentation](https://mlsanity.com/paper/2307.08587/) (2023)
- [Multilingual Standalone Trustworthy Voice-Based Social Network for Disaster Situations](https://mlsanity.com/paper/2411.08889/) (2024)
- [Understanding Language Modeling Paradigm Adaptations in Recommender Systems: Lessons Learned and Open Challenges](https://mlsanity.com/paper/2404.03788/) (2024)

## Built on by (most cited first, 2 of 2)

- [When Generative Artificial Intelligence meets Extended Reality: A Systematic Review](https://mlsanity.com/paper/2511.03282/) (2025)
- [Towards user-centered interactive medical image segmentation in VR with an assistive AI agent](https://mlsanity.com/paper/2505.07214/) (2025)

## Abstract (arXiv)

This paper presents Matrix, an advanced AI-powered framework designed for real-time 3D object generation in Augmented Reality (AR) environments. By integrating a cutting-edge text-to-3D generative AI model, multilingual speech-to-text translation, and large language models (LLMs), the system enables seamless user interactions through spoken commands. The framework processes speech inputs, generates 3D objects, and provides object recommendations based on contextual understanding, enhancing AR experiences. A key feature of this framework is its ability to optimize 3D models by reducing mesh complexity, resulting in significantly smaller file sizes and faster processing on resource-constrained AR devices. Our approach addresses the challenges of high GPU usage, large model output sizes, and real-time system responsiveness, ensuring a smoother user experience. Moreover, the system is equipped with a pre-generated object repository, further reducing GPU load and improving efficiency. We demonstrate the practical applications of this framework in various fields such as education, design, and accessibility, and discuss future enhancements including image-to-3D conversion, environmental object detection, and multimodal support. The open-source nature of the framework promotes ongoing innovation and its utility across diverse industries.