mlsanityData to 7 Oct 2026

SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos

arXiv 2404.05206 ↗First posted 8 Apr 2024cs.CV cs.MM cs.SDcited by 11

Not on the map. Why?

Abstract (arXiv)

We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos. Whereas existing methods rely on curated data with known audio-visual correspondence, our multimodal contrastive-consensus coding (MC3) embedding reinforces the associations between audio, language, and vision when all modality pairs agree, while diminishing those associations when any one pair does not. We show our approach can successfully discover how the long tail of human actions sound from egocentric video, outperforming an array of recent multimodal embedding techniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modal tasks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 44 of its 57 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

44 of 57 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  12. 2021AST: Audio Spectrogram Transformercited by 634
32 more
  1. 2019On Variational Bounds of Mutual Informationcited by 550
  2. 2017Look, Listen and Learncited by 452
  3. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  4. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  5. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  6. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  7. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  8. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  9. 2017Objects that Soundcited by 254
  10. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  11. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  12. 2022Egocentric Video-Language Pretrainingcited by 231
  13. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  14. 2022Learning Video Representations from Large Language Modelscited by 204
  15. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  16. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  17. 2015Visually Indicated Soundscited by 128
  18. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  19. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  20. 2021Robust Audio-Visual Instance Discriminationcited by 80
  21. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  22. 20182.5D Visual Soundcited by 63
  23. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  24. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  25. 2021Audio Captioning Transformercited by 53
  26. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  27. 2022Visual Acoustic Matchingcited by 45
  28. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  29. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  30. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  31. 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19
  32. 2020Swoosh! Rattle! Thump! -- Actions that Soundcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 1.

11 papers.

  1. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  2. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  3. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  4. 2024EgoSonics: Generating Synchronized Audio for Silent Egocentric Videoscited by 6
  5. 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generationcited by 5
  6. 2025Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidancecited by 2
  7. 2025Sonify Anything: Towards Context-Aware Sonic Interactions in ARcited by 1
  8. 2024ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgationcited by 0
  9. 2025CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognitioncited by 0
  10. 2025Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactionscited by 0
  11. 2026Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videoscited by 0