# Augmented Object Intelligence with XR-Objects

arXiv: https://arxiv.org/abs/2404.13274 · HTML: https://mlsanity.com/paper/2404.13274/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 94 references that are in the map.
- First posted on arXiv 20 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation](https://mlsanity.com/paper/2310.12953/) (2023)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023)
- [RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching](https://mlsanity.com/paper/2008.08688/) (2020)
- [InfraredTags: Embedding Invisible AR Markers and Barcodes Using Low-Cost, Infrared-Based 3D Printing and Imaging Tools](https://mlsanity.com/paper/2202.06165/) (2022)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [The Dark Side of Perceptual Manipulations in Virtual Reality](https://mlsanity.com/paper/2202.13200/) (2022)
- [Memory Manipulations in Extended Reality](https://mlsanity.com/paper/2304.02394/) (2023)
- [Augmenting Static Visualizations with PapARVis Designer](https://mlsanity.com/paper/2310.04826/) (2023)
- [ARShopping: In-Store Shopping Decision Support Through Augmented Reality and Immersive Visualization](https://mlsanity.com/paper/2207.07643/) (2022)
- [HoloTouch: Interacting with Mixed Reality Visualizations Through Smartphone Proxies](https://mlsanity.com/paper/2303.08916/) (2023)
- [Understanding Physical Breakdowns in Virtual Reality](https://mlsanity.com/paper/2404.00025/) (2024)

## Built on by (most cited first, 12 of 12)

- [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025)
- [VisionClaw: Always-On AI Agents through Smart Glasses](https://mlsanity.com/paper/2604.03486/) (2026)
- [PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos](https://mlsanity.com/paper/2407.09503/) (2024)
- [SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Reality](https://mlsanity.com/paper/2602.00793/) (2026)
- [Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering](https://mlsanity.com/paper/2511.11930/) (2025)
- [Accelerating Physical Property Reasoning for Augmented Visual Cognition](https://mlsanity.com/paper/2511.03126/) (2025)
- [PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Cases](https://mlsanity.com/paper/2512.17172/) (2025)
- [Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant](https://mlsanity.com/paper/2601.17622/) (2026)
- [From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality](https://mlsanity.com/paper/2602.03059/) (2026)
- [RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtion](https://mlsanity.com/paper/2410.13412/) (2024)
- [Imprinto: Enhancing Infrared Inkjet Watermarking for Human and Machine Perception](https://mlsanity.com/paper/2502.17089/) (2025)
- [XR Blocks: Accelerating Human-centered AI + XR Innovation](https://mlsanity.com/paper/2509.25504/) (2025)

## Abstract (arXiv)

Seamless integration of physical objects as interactive digital entities remains a challenge for spatial computing. This paper explores Augmented Object Intelligence (AOI) in the context of XR, an interaction paradigm that aims to blur the lines between digital and physical by equipping real-world objects with the ability to interact as if they were digital, where every object has the potential to serve as a portal to digital functionalities. Our approach utilizes real-time object segmentation and classification, combined with the power of Multimodal Large Language Models (MLLMs), to facilitate these interactions without the need for object pre-registration. We implement the AOI concept in the form of XR-Objects, an open-source prototype system that provides a platform for users to engage with their physical environment in contextually relevant ways using object-based context menus. This system enables analog objects to not only convey information but also to initiate digital actions, such as querying for details or executing tasks. Our contributions are threefold: (1) we define the AOI concept and detail its advantages over traditional AI assistants, (2) detail the XR-Objects system's open-source design and implementation, and (3) show its versatility through various use cases and a user study.