# EmBARDiment: an Embodied AI Agent for Productivity in XR

arXiv: https://arxiv.org/abs/2408.08158 · HTML: https://mlsanity.com/paper/2408.08158/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 50 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 7 of 7)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Automatic Generation of Chatbots for Conversational Web Browsing](https://mlsanity.com/paper/2008.12097/) (2020)
- [Text Entry in Immersive Head-Mounted Display-based Virtual Reality using Standard Keyboards](https://mlsanity.com/paper/1802.00626/) (2018)
- [Breaking the Screen: Interaction Across Touchscreen Boundaries in Virtual Reality for Mobile Knowledge Workers](https://mlsanity.com/paper/2008.04559/) (2020)
- [Towards a Practical Virtual Office for Mobile Knowledge Workers](https://mlsanity.com/paper/2009.02947/) (2020)

## Built on by (most cited first, 7 of 7)

- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR](https://mlsanity.com/paper/2506.05683/) (2025)
- [OpenGlass: Ultra-Low-Power On-Device AI Eyewear with Event-based Vision](https://mlsanity.com/paper/2606.07431/) (2026)
- [A cybersecurity AI agent selection and decision support framework](https://mlsanity.com/paper/2510.01751/) (2025)
- [Reality Copilot: Voice-First Human-AI Collaboration in Mixed Reality Using Large Multimodal Models](https://mlsanity.com/paper/2602.11025/) (2026)
- [From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants](https://mlsanity.com/paper/2604.08062/) (2026)
- [How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems](https://mlsanity.com/paper/2608.07861/) (2026)

## Abstract (arXiv)

XR devices running chat-bots powered by Large Language Models (LLMs) have the to become always-on agents that enable much better productivity scenarios. Current screen based chat-bots do not take advantage of the the full-suite of natural inputs available in XR, including inward facing sensor data, instead they over-rely on explicit voice or text prompts, sometimes paired with multi-modal data dropped as part of the query. We propose a solution that leverages an attention framework that derives context implicitly from user actions, eye-gaze, and contextual memory within the XR environment. Our work minimizes the need for engineered explicit prompts, fostering grounded and intuitive interactions that glean user insights for the chat-bot.