# RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models

arXiv: https://arxiv.org/abs/2405.18620 · HTML: https://mlsanity.com/paper/2405.18620/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 66 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization](https://mlsanity.com/paper/2110.07356/) (2021)
- [Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries](https://mlsanity.com/paper/2208.09323/) (2022)
- [Prompted Opinion Summarization with GPT-3.5](https://mlsanity.com/paper/2211.15914/) (2022)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols](https://mlsanity.com/paper/2009.14237/) (2020)
- [CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Context](https://mlsanity.com/paper/2302.07302/) (2023)
- [Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections](https://mlsanity.com/paper/2302.06754/) (2023)
- [Conversations with Documents. An Exploration of Document-Centered Assistance](https://mlsanity.com/paper/2002.00747/) (2020)
- [ComLittee: Literature Discovery with Personal Elected Author Committees](https://mlsanity.com/paper/2302.06780/) (2023)
- [G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios](https://mlsanity.com/paper/2405.07652/) (2024)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summaries](https://mlsanity.com/paper/2303.17826/) (2023)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [Augmenting Static Visualizations with PapARVis Designer](https://mlsanity.com/paper/2310.04826/) (2023)
- [SummaryLens -- A Smartphone App for Exploring Interactive Use of Automated Text Summarization in Everyday Life](https://mlsanity.com/paper/2202.02053/) (2022)

## Abstract (arXiv)

Large Language Models (LLMs) are gaining popularity as reading and summarization aids. However, little is known about their potential benefits when integrated with mixed reality (MR) interfaces to support everyday reading. In this iterative investigation, we developed RealitySummary, an MR reading assistant that seamlessly integrates LLMs with always-on camera access, OCR-based text extraction, and augmented spatial and visual responses. Developed iteratively, RealitySummary evolved across three versions, each shaped by user feedback and reflective analysis: 1) a preliminary user study to understand reader perceptions (N=12), 2) an in-the-wild deployment to explore real-world usage (N=11), and 3) a diary study to capture insights from real-world work contexts (N=5). Our empirical studies' findings highlight the unique advantages of combining AI and MR, including always-on implicit assistance, long-term temporal history, minimal context switching, and spatial affordances, demonstrating significant potential for future LLM-MR interfaces beyond traditional screen-based interactions.