# Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations

arXiv: https://arxiv.org/abs/2507.17248 · HTML: https://mlsanity.com/paper/2507.17248/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 108 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Builds on (most cited first, 18 of 18)

- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [iBall: Augmenting Basketball Videos with Gaze-moderated Embedded Visualizations](https://mlsanity.com/paper/2303.03476/) (2023)
- [CrossData: Leveraging Text-Data Connections for Authoring Data Documents](https://mlsanity.com/paper/2310.11639/) (2023)
- [Augmenting Static Visualizations with PapARVis Designer](https://mlsanity.com/paper/2310.04826/) (2023)
- [CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision](https://mlsanity.com/paper/2407.13515/) (2024)
- [HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots](https://mlsanity.com/paper/2108.10829/) (2021)
- [StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible](https://mlsanity.com/paper/1908.07144/) (2019)
- [VisiMark: Characterizing and Augmenting Landmarks for People with Low Vision in Augmented Reality to Support Indoor Navigation](https://mlsanity.com/paper/2502.10561/) (2025)
- [PinchCatcher: Enabling Multi-selection for Gaze+Pinch](https://mlsanity.com/paper/2503.05456/) (2025)
- [Symbiotic AI: Augmenting Human Cognition from PCs to Cars](https://mlsanity.com/paper/2504.03105/) (2025)
- [Exploring Interactions with Printed Data Visualizations in Augmented Reality](https://mlsanity.com/paper/2208.10603/) (2022)

## Built on by (most cited first, 1 of 1)

- [From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality](https://mlsanity.com/paper/2602.03059/) (2026)

## Abstract (arXiv)

Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.