# PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Cases

arXiv: https://arxiv.org/abs/2512.17172 · HTML: https://mlsanity.com/paper/2512.17172/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS](https://mlsanity.com/paper/2304.00501/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [LLMR: Real-time Prompting of Interactive Worlds using Large Language Models](https://mlsanity.com/paper/2309.12276/) (2023)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023)
- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence](https://mlsanity.com/paper/2106.08710/) (2021)
- [G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios](https://mlsanity.com/paper/2405.07652/) (2024)
- [Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8](https://mlsanity.com/paper/2306.03537/) (2023)
- [TOM: A Development Platform For Wearable Intelligent Assistants](https://mlsanity.com/paper/2407.15523/) (2024)
- [Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality](https://mlsanity.com/paper/2404.13777/) (2024)

## Built on by (most cited first, 1 of 1)

- [SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Reality](https://mlsanity.com/paper/2602.00793/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI)-driven augmented reality (AR) systems are becoming increasingly integrated into daily life, and with this growth comes a greater need for explainability in real-time user interactions. Traditional explainable AI (XAI) methods, which often rely on feature-based or example-based explanations, struggle to deliver dynamic, context-specific, personalized, and human-centric insights for everyday AR users. These methods typically address separate explainability dimensions (e.g., when, what, how) with different explanation techniques, resulting in unrealistic and fragmented experiences for seamless AR interactions. To address this challenge, we propose PILAR, a novel framework that leverages a pre-trained large language model (LLM) to generate context-aware, personalized explanations, offering a more intuitive and trustworthy experience in real-time AI-powered AR systems. Unlike traditional methods, which rely on multiple techniques for different aspects of explanation, PILAR employs a unified LLM-based approach that dynamically adapts explanations to the user's needs, fostering greater trust and engagement. We implement the PILAR concept in a real-world AR application (e.g., personalized recipe recommendations), an open-source prototype that integrates real-time object detection, recipe recommendation, and LLM-based personalized explanations of the recommended recipes based on users' dietary preferences. We evaluate the effectiveness of PILAR through a user study with 16 participants performing AR-based recipe recommendation tasks, comparing an LLM-based explanation interface to a traditional template-based one. Results show that the LLM-based interface significantly enhances user performance and experience, with participants completing tasks 40% faster and reporting greater satisfaction, ease of use, and perceived transparency.