mlsanityData to 7 Oct 2026

Spherical World-Locking for Audio-Visual Localization in Egocentric Videos

arXiv 2408.05364 ↗First posted 9 Aug 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Egocentric videos provide comprehensive contexts for user and scene understanding, spanning multisensory perception to behavioral interaction. We propose Spherical World-Locking (SWL) as a general framework for egocentric scene representation, which implicitly transforms multisensory streams with respect to measurements of head orientation. Compared to conventional head-locked egocentric representations with a 2D planar field-of-view, SWL effectively offsets challenges posed by self-motion, allowing for improved spatial synchronization between input modalities. Using a set of multisensory embeddings on a worldlocked sphere, we design a unified encoder-decoder transformer architecture that preserves the spherical structure of the scene representation, without requiring expensive projections between image and world coordinate systems. We evaluate the effectiveness of the proposed framework on multiple benchmark tasks for egocentric video understanding, including audio-visual active speaker localization, auditory spherical source localization, and behavior anticipation in everyday activities.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 60 of its 80 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

60 of 80 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021Vision Transformers for Dense Predictioncited by 1,789
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2015Spatial Transformer Networkscited by 1,423
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2021AST: Audio Spectrogram Transformercited by 634
  12. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
38 more
  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2018Spherical CNNscited by 452
  3. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  4. 2018The Sound of Pixelscited by 258
  5. 2017Objects that Soundcited by 254
  6. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 244
  7. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  8. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  9. 2019The Sound of Motionscited by 145
  10. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  11. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  12. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  13. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  14. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  15. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  16. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
  17. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  18. 2019Tangent Images for Mitigating Spherical Distortioncited by 79
  19. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  20. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  21. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  22. 2022In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimationcited by 51
  23. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 46
  24. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  25. 2016Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Videocited by 45
  26. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  27. 2023Egocentric Audio-Visual Object Localizationcited by 40
  28. 2022IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Textcited by 36
  29. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  30. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  31. 2018A Memory Network Approach for Story-based Temporal Summarization of 360deg Videoscited by 29
  32. 2021EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environmentscited by 29
  33. 2023Balanced Spherical Grid for Egocentric View Synthesiscited by 29
  34. 2021Egocentric Pose Estimation from Human Vision Spancited by 27
  35. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  36. 2022EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queriescited by 25
  37. 2016Pano2Vid: Automatic Cinematography for Watching 360circ Videoscited by 22
  38. 2022EgoEnv: Human-centric environment representations from egocentric videocited by 22

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  2. 2025Decoupled Audio-Visual Dataset Distillationcited by 6
  3. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  4. 2025Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Spancited by 2
  5. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0