mlsanityData to 7 Oct 2026

Listen to Look: Action Recognition by Previewing Audio

arXiv 1912.04487 ↗First posted 10 Dec 2019cs.CV cs.LG cs.SDcited by 163

Not on the map. Why?

Abstract (arXiv)

In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed video that uses audio as a preview mechanism to eliminate both short-term and long-term visual redundancies. First, we devise an ImgAud2Vid framework that hallucinates clip-level features by distilling from lighter modalities---a single frame and its accompanying audio---reducing short-term temporal redundancy for efficient clip-level recognition. Second, building on ImgAud2Vid, we further propose ImgAud-Skimming, an attention-based long short-term memory network that iteratively selects useful moments in untrimmed videos, reducing long-term temporal redundancy for efficient video-level recognition. Extensive experiments on four action recognition datasets demonstrate that our method achieves the state-of-the-art in terms of both recognition accuracy and speed.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 46; 2023: 37; 2024: 27; 2025: 17; 2026: 6.

163 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2021Dynamic Neural Networks: A Surveycited by 405
  3. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  4. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  5. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  6. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  7. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  8. 2021Adaptive Focus for Efficient Video Recognitioncited by 76
  9. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  10. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  11. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  12. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
38 more
  1. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  2. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  3. 2022Dynamic Multimodal Fusioncited by 54
  4. 2021MGSampler: An Explainable Sampling Strategy for Video Action Recognitioncited by 49
  5. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  6. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48
  7. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  8. 2021MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognitioncited by 44
  9. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  10. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  11. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  12. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  13. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  14. 2023A Light Weight Model for Active Speaker Detectioncited by 39
  15. 2021AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognitioncited by 37
  16. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  17. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  18. 2021Dynamic Network Quantization for Efficient Video Inferencecited by 36
  19. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  20. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  21. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  22. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  23. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  24. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  25. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 30
  26. 2022Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsingcited by 29
  27. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  28. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  29. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  30. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 24
  31. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  32. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  33. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  34. 2022OCSampler: Compressing Videos to One Clip with Single-step Samplingcited by 22
  35. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  36. 2023Physics-Driven Diffusion Models for Impact Sound Synthesis from Videoscited by 21
  37. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  38. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20

50 most cited of 163 All 163 (JSON)