mlsanityData to 14 Oct 2026

ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound

arXiv 2204.02874 ↗First posted 6 Apr 2022cs.CV cs.AI cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

We introduce an audiovisual method for long-range text-to-video retrieval. Unlike previous approaches designed for short video retrieval (e.g., 5-15 seconds in duration), our approach aims to retrieve minute-long videos that capture complex human actions. One challenge of standard video-only approaches is the large computational cost associated with processing hundreds of densely extracted frames from such long videos. To address this issue, we propose to replace parts of the video with compact audio cues that succinctly summarize dynamic audio events and are cheap to process. Our method, named ECLIPSE (Efficient CLIP with Sound Encoding), adapts the popular CLIP model to an audiovisual video setting, by adding a unified audiovisual transformer block that captures complementary cues from the video and audio streams. In addition to being 2.92x faster and 2.34x memory-efficient than long-range video-only approaches, our method also achieves better text-to-video retrieval accuracy on several diverse long-range video datasets such as ActivityNet, QVHighlights, YouCook2, DiDeMo and Charades.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 77 of its 89 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

77 of 89 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
38 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  2. 2017Dense-Captioning Events in Videoscited by 793
  3. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  4. 2017Localizing Moments in Video with Natural Languagecited by 663
  5. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  7. 2021AST: Audio Spectrogram Transformercited by 643
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  10. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  13. 2017Look, Listen and Learncited by 458
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  15. 2020Multi-modal Transformer for Video Retrievalcited by 395
  16. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  17. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  18. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  19. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  20. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  21. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  22. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  23. 2017Objects that Soundcited by 256
  24. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  25. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  26. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  27. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  28. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  29. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  30. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  31. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  32. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  33. 2020Support-set bottlenecks for video-text representation learningcited by 170
  34. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  35. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 128
  36. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  37. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  38. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 10; 2024: 10; 2025: 7; 2026: 4.

39 papers.

Sort:

By area:

  1. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  2. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  3. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  4. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  5. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  6. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  7. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  8. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  9. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  10. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  11. 2024Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrievalcited by 33
  12. 2022CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Groundingcited by 27
27 more
  1. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  2. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25
  3. 2022TVLT: Textless Vision-Language Transformercited by 22
  4. 2023Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignmentcited by 21
  5. 2023Multiscale Audio Spectrogram Transformer for Efficient Audio Classificationcited by 16
  6. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  7. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 14
  8. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  9. 2023CAST: Cross-Attention in Space and Time for Video Action Recognitioncited by 12
  10. 2023SViTT: Temporal Learning of Sparse Video-Text Transformerscited by 11
  11. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
  12. 2024MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learningcited by 8
  13. 2025Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrievalcited by 8
  14. 2024Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metriccited by 6
  15. 2024Towards Long Video Understanding via Fine-detailed Video Story Generationcited by 6
  16. 2025NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrievalcited by 6
  17. 2022MuMUR : Multilingual Multimodal Universal Retrievalcited by 3
  18. 2024Activity Recognition on Avatar-Anonymized Datasets with Masked Differential Privacycited by 3
  19. 2026CoVA: Text-Guided Composed Video Retrieval for Audio-Visual Contentcited by 2
  20. 2025Leveraging Modality Tags for Enhanced Cross-Modal Video Retrievalcited by 1
  21. 2025Leveraging Auxiliary Information in Text-to-Video Retrieval: A Reviewcited by 1
  22. 2026Sounding Highlights: Dual-Pathway Audio Encoders for Audio-Visual Video Highlight Detectioncited by 1
  23. 2023Query-aware Long Video Localization and Relation Discrimination for Deep Video Understandingcited by 0
  24. 2025GAIS: Frame-Level Gated Audio-Visual Integration with Semantic Variance-Scaled Perturbation for Text-Video Retrievalcited by 0
  25. 2025FDRMFL: Multimodal Federated Feature Extraction Model Based on Information Maximization and Contrastive Learningcited by 0
  26. 2026SAVE: Speech-Aware Video Representation Learning for Video-Text Retrievalcited by 0
  27. 2026Text-to-CAD Retrieval: a Strong Baselinecited by 0