mlsanityData to 7 Oct 2026

OWL (Observe, Watch, Listen): Audiovisual Temporal Context for Localizing Actions in Egocentric Videos

arXiv 2202.04947 ↗First posted 10 Feb 2022cs.CV cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Egocentric videos capture sequences of human activities from a first-person perspective and can provide rich multimodal signals. However, most current localization methods use third-person videos and only incorporate visual information. In this work, we take a deep look into the effectiveness of audiovisual context in detecting actions in egocentric videos and introduce a simple-yet-effective approach via Observing, Watching, and Listening (OWL). OWL leverages audiovisual information and context for egocentric temporal action localization (TAL). We validate our approach in two large-scale datasets, EPIC-Kitchens, and HOMAGE. Extensive experiments demonstrate the relevance of the audiovisual temporal context. Namely, we boost the localization performance (mAP) over visual-only models by +2.23% and +3.35% in the above datasets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2021ViViT: A Video Vision Transformercited by 1,610
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2020Rescaling Egocentric Visioncited by 529
  12. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
31 more
  1. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  2. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  3. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  4. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  5. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  6. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  7. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  8. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  9. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  10. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  11. 2021Anticipative Video Transformercited by 197
  12. 2017R-C3D: Region Convolutional 3D Network for Temporal Activity Detectioncited by 161
  13. 2021End-to-end Temporal Action Detection with Transformercited by 150
  14. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  15. 2021Temporal Context Aggregation Network for Temporal Action Proposal Refinementcited by 111
  16. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  17. 2020Video Self-Stitching Graph Network for Temporal Action Localizationcited by 104
  18. 2020Boundary Content Graph Neural Network for Temporal Action Proposal Generationcited by 88
  19. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  20. 2017Learnable pooling with Context Gating for video classificationcited by 58
  21. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  22. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 52
  23. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48
  24. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  25. 2017Temporal Context Network for Activity Localization in Videoscited by 37
  26. 2021Class Semantics-based Attention for Action Detectioncited by 36
  27. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
  28. 2021Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localizationcited by 21
  29. 2019Human Action Sequence Classificationcited by 3
  30. 2021Modeling long-term interactions to enhance action recognitioncited by 3
  31. 2021A Stronger Baseline for Ego-Centric Action Detectioncited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  2. 2024Test-Time Adaptation for Combating Missing Modalities in Egocentric Videoscited by 2
  3. 2023Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localizationcited by 1
  4. 2025DEL: Dense Event Localization for Multi-modal Audio-Visual Understandingcited by 1