A Benchmarking Framework for Context-aware XR Interfaces
Not on the map. Why?
Abstract (arXiv)
Everyday Extended Reality (XR) systems aim to provide context-aware access to the right functionalities at the right time and place, with minimal manual reconfiguration as users switch context. Yet these interfaces are hard to evaluate: current prototyping and user-study workflows offer no systematic, repeatable way to compare adaptation methods across users and scenarios. We present ContextXR, a novel benchmarking framework for context-aware XR interfaces. ContextXR represents an XR application as a connected graph of functional facets, each a semantically coherent group of related capabilities that together support a shared user intent. On this representation, we build MineXR++, a dataset augmenting prior XR interface data with facet-level annotations, and formulate three canonical tasks of context-aware suggestion: context factor analysis, initial facet suggestion, and next facet suggestion. Our evaluation protocol scores suggestion methods by a simulated interaction metric, the navigation and search cost of reaching the desired functionality. Through experiments benchmarking global popularity, relational retrieval, and LLM-based methods, we demonstrate that ContextXR enables the systematic, reproducible evaluation of context-aware XR interfaces.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 70 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
27 of 70 references.
References · 27
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
- 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
- 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 45
- 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
15 more
- 2021VINS: Visual Search for Mobile User Interface Designcited by 37
- 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
- 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
- 2025Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Modelcited by 26
- 2024Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modelingcited by 25
- 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
- 2021Adapting User Interfaces with Model-based Reinforcement Learningcited by 17
- 2025Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agentscited by 15
- 2024OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMscited by 10
- 2024SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM Reasoningcited by 8
- 2025Generative Interfaces for Language Modelscited by 6
- 2022Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinementscited by 2
- 2024MineXR: Mining Personalized Extended Reality Interfacescited by 1
- 2026Log2Motion: Biomechanical Motion Synthesis from Touch Logscited by 1
- 2026Automating UI Optimization through Multi-Agentic Reasoningcited by 1
Built on by
Not cited yet.