# Reading Recognition in the Wild

arXiv: https://arxiv.org/abs/2505.24848 · HTML: https://mlsanity.com/paper/2505.24848/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Integrating Human Gaze into Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/2011.03920/) (2020)
- [GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear](https://mlsanity.com/paper/2401.17217/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patterns](https://mlsanity.com/paper/2404.02370/) (2024)
- [Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image](https://mlsanity.com/paper/2406.14050/) (2024)
- [GazeHTA: End-to-end Gaze Target Detection with Head-Target Association](https://mlsanity.com/paper/2404.10718/) (2024)
- [OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction](https://mlsanity.com/paper/2407.13335/) (2024)
- [Look Hear: Gaze Prediction for Speech-directed Human Attention](https://mlsanity.com/paper/2407.19605/) (2024)
- [Involution Fused ConvNet for Classifying Eye-Tracking Patterns of Children with Autism Spectrum Disorder](https://mlsanity.com/paper/2401.03575/) (2024)
- [Learning User Embeddings from Human Gaze for Personalised Saliency Prediction](https://mlsanity.com/paper/2403.13653/) (2024)
- [SARA: Smart AI Reading Assistant for Reading Comprehension](https://mlsanity.com/paper/2404.06906/) (2024)
- [GazePrompt: Enhancing Low Vision People's Reading Experience with Gaze-Aware Augmentations](https://mlsanity.com/paper/2402.12772/) (2024)
- [A Transformer-Based Model for the Prediction of Human Gaze Behavior on Videos](https://mlsanity.com/paper/2404.07351/) (2024)
- [Gaze-DETR: Using Expert Gaze to Reduce False Positives in Vulvovaginal Candidiasis Screening](https://mlsanity.com/paper/2405.09463/) (2024)
- [Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model](https://mlsanity.com/paper/2408.01044/) (2024)
- [GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph](https://mlsanity.com/paper/2408.05502/) (2024)

## Built on by (most cited first, 6 of 6)

- [Seeing without Pixels: Perception from Camera Trajectories](https://mlsanity.com/paper/2511.21681/) (2025)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory](https://mlsanity.com/paper/2606.00825/) (2026)
- [Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling](https://mlsanity.com/paper/2608.22926/) (2026)
- [Will You Be Aware? Eye Tracking-Based Modeling of Situational Awareness in Augmented Reality](https://mlsanity.com/paper/2508.05025/) (2025)
- [OmniEgoCap: Camera-Agnostic Sequence-Level Egocentric Motion Reconstruction](https://mlsanity.com/paper/2512.19283/) (2025)

## Abstract (arXiv)

To enable egocentric contextual AI in always-on smart glasses, it is crucial to be able to keep a record of the user's interactions with the world, including during reading. In this paper, we introduce a new task of reading recognition to determine when the user is reading. We first introduce the first-of-its-kind large-scale multimodal Reading in the Wild dataset, containing 100 hours of reading and non-reading videos in diverse and realistic scenarios. We then identify three modalities (egocentric RGB, eye gaze, head pose) that can be used to solve the task, and present a flexible transformer model that performs the task using these modalities, either individually or combined. We show that these modalities are relevant and complementary to the task, and investigate how to efficiently and effectively encode each modality. Additionally, we show the usefulness of this dataset towards classifying types of reading, extending current reading understanding studies conducted in constrained settings to larger scale, diversity and realism.