mlsanityData to 7 Oct 2026

Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos

arXiv 2307.04760 ↗First posted 10 Jul 2023cs.CV cs.SD eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a self-supervised method for learning representations based on spatial audio-visual correspondences in egocentric videos. Our method uses a masked auto-encoding framework to synthesize masked binaural (multi-channel) audio through the synergy of audio and vision, thereby learning useful spatial relationships between the two modalities. We use our pretrained features to tackle two downstream video tasks requiring spatial understanding in social scenarios: active speaker detection and spatial audio denoising. Through extensive experiments, we show that our features are generic enough to improve over multiple state-of-the-art baselines on both tasks on two challenging egocentric video datasets that offer binaural audio, EgoCom and EasyCom. Project: http://vision.cs.utexas.edu/projects/ego_av_corr.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 58 of its 84 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

58 of 84 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2016Layer Normalizationcited by 5,579
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2020Rescaling Egocentric Visioncited by 529
38 more
  1. 2017Look, Listen and Learncited by 452
  2. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  3. 2018SDR - half-baked or well done?cited by 420
  4. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  5. 2022Masked Autoencoders that Listencited by 271
  6. 2023Multimodal Deep Learningcited by 257
  7. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  8. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  9. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  10. 2015Visually Indicated Soundscited by 128
  11. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  12. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  13. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  14. 2019Co-Separating Sounds of Visual Objectscited by 110
  15. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  16. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  17. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  18. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  19. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  20. 2022Localizing Visual Sounds the Easy Waycited by 81
  21. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  22. 20182.5D Visual Soundcited by 63
  23. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  24. 2022MAViL: Masked Audio-Video Learnerscited by 56
  25. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  26. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  27. 2022Audiovisual Masked Autoencoderscited by 47
  28. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  29. 2022Visual Acoustic Matchingcited by 45
  30. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  31. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  32. 2020Active Speakers in Contextcited by 40
  33. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
  34. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  35. 2020Audio-Visual Floorplan Reconstructioncited by 32
  36. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 30
  37. 2021EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environmentscited by 29
  38. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 29

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  2. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  3. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  4. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  5. 2024EgoSonics: Generating Synchronized Audio for Silent Egocentric Videoscited by 6
  6. 2025ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localizationcited by 1
  7. 2025Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Videocited by 0
  8. 2026A Framework for Egocentric and Exocentric Procedural Understanding via Temporal Segmentation and Semantic Abstractioncited by 0