# Seeing and Hearing Egocentric Actions: How Much Can We Learn?

arXiv: https://arxiv.org/abs/1910.06693 · HTML: https://mlsanity.com/paper/1910.06693/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition](https://mlsanity.com/paper/2112.03596/) (2021)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos](https://mlsanity.com/paper/2203.03014/) (2022)
- [Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment](https://mlsanity.com/paper/2106.01689/) (2021)
- [Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries](https://mlsanity.com/paper/2102.08065/) (2021)
- [Bringing Online Egocentric Action Recognition into the wild](https://mlsanity.com/paper/2211.03004/) (2022)
- [The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning](https://mlsanity.com/paper/2110.07082/) (2021)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)

## Abstract (arXiv)

Our interaction with the world is an inherently multimodal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial, and temporal streams. Experimental results on the EPIC-Kitchens dataset show that multimodal integration leads to better performance than unimodal approaches. In particular, we achieved a 5.18% improvement over the state of the art on verb classification.