# EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments

arXiv: https://arxiv.org/abs/2107.04174 · HTML: https://mlsanity.com/paper/2107.04174/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 12 of 12)

- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric](https://mlsanity.com/paper/2004.09584/) (2020)
- [DiPCo -- Dinner Party Corpus](https://mlsanity.com/paper/1909.13447/) (2019)
- [An instrumental intelligibility metric based on information theory](https://mlsanity.com/paper/1708.05132/) (2017)
- [An evaluation of intrusive instrumental intelligibility metrics](https://mlsanity.com/paper/1708.06027/) (2017)

## Built on by (most cited first, 30 of 31)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding](https://mlsanity.com/paper/2406.08877/) (2024)
- [Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization](https://mlsanity.com/paper/2201.01928/) (2022)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement](https://mlsanity.com/paper/2212.11377/) (2022)
- [Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos](https://mlsanity.com/paper/2307.04760/) (2023)
- [Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning](https://mlsanity.com/paper/2308.09514/) (2023)
- [EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild](https://mlsanity.com/paper/2502.14892/) (2025)
- [WASD: A Wilder Active Speaker Detection Dataset](https://mlsanity.com/paper/2303.05321/) (2023)
- [Spherical World-Locking for Audio-Visual Localization in Egocentric Videos](https://mlsanity.com/paper/2408.05364/) (2024)
- [BIAS: A Body-based Interpretable Active Speaker Approach](https://mlsanity.com/paper/2412.05150/) (2024)
- [ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection](https://mlsanity.com/paper/2412.08594/) (2024)
- [EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception](https://mlsanity.com/paper/2506.21080/) (2025)
- [Self-supervised Video-centralised Transformer for Video Face Clustering](https://mlsanity.com/paper/2203.13166/) (2022)
- [Inclusivity of AI Speech in Healthcare: A Decade Look Back](https://mlsanity.com/paper/2505.10596/) (2025)
- [Direction-Aware Adaptive Online Neural Speech Enhancement with an Augmented Reality Headset in Real Noisy Conversational Environments](https://mlsanity.com/paper/2207.07296/) (2022)
- [LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2406.04432/) (2024)
- [Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment](https://mlsanity.com/paper/2501.18157/) (2025)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)
- [Direction-Aware Joint Adaptation of Neural Speech Enhancement and Recognition in Real Multiparty Conversational Environments](https://mlsanity.com/paper/2207.07273/) (2022)
- [Audio-Visual Talker Localization in Video for Spatial Sound Reproduction](https://mlsanity.com/paper/2406.00495/) (2024)
- [Identification of Conversation Partners from Egocentric Video](https://mlsanity.com/paper/2406.08089/) (2024)
- [FabuLight-ASD: Unveiling Speech Activity via Body Language](https://mlsanity.com/paper/2411.13674/) (2024)
- [SHAMaNS: Sound Localization with Hybrid Alpha-Stable Spatial Measure and Neural Steerer](https://mlsanity.com/paper/2506.18954/) (2025)
- [Gaussian Process Regression of Steering Vectors With Physics-Aware Deep Composite Kernels for Augmented Listening](https://mlsanity.com/paper/2509.02571/) (2025)
- [When Forgetting Looks Like Improvement: Metric Masking in Streaming Diarizer Adaptation and the Price of Rehearsal](https://mlsanity.com/paper/2610.08828/) (2026)

## Abstract (arXiv)

Augmented Reality (AR) as a platform has the potential to facilitate the reduction of the cocktail party effect. Future AR headsets could potentially leverage information from an array of sensors spanning many different modalities. Training and testing signal processing and machine learning algorithms on tasks such as beam-forming and speech enhancement require high quality representative data. To the best of the author's knowledge, as of publication there are no available datasets that contain synchronized egocentric multi-channel audio and video with dynamic movement and conversations in a noisy environment. In this work, we describe, evaluate and release a dataset that contains over 5 hours of multi-modal data useful for training and testing algorithms for the application of improving conversations for an AR glasses wearer. We provide speech intelligibility, quality and signal-to-noise ratio improvement results for a baseline method and show improvements across all tested metrics. The dataset we are releasing contains AR glasses egocentric multi-channel microphone array audio, wide field-of-view RGB video, speech source pose, headset microphone audio, annotated voice activity, speech transcriptions, head bounding boxes, target of speech and source identification labels. We have created and are releasing this dataset to facilitate research in multi-modal AR solutions to the cocktail party problem.