mlsanityData to 7 Oct 2026

MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval

arXiv 2408.10575 ↗First posted 20 Aug 2024cs.CVcited by 7

On the map: State-space sequence models (S4, Mamba) → Video, motion and generative Mamba models

Abstract (arXiv)

Text-Video Retrieval (TVR) aims to align and associate relevant video content with corresponding natural language queries. Most existing TVR methods are based on large-scale pre-trained vision-language models (e.g., CLIP). However, due to the inherent plain structure of CLIP, few TVR methods explore the multi-scale representations which offer richer contextual information for a more thorough understanding. To this end, we propose MUSE, a multi-scale mamba with linear computational complexity for efficient cross-resolution modeling. Specifically, the multi-scale representations are generated by applying a feature pyramid on the last single-scale feature map. Then, we employ the Mamba structure as an efficient multi-scale learner to jointly learn scale-wise representations. Furthermore, we conduct comprehensive studies to investigate different model structures and designs. Extensive results on three popular benchmarks have validated the superiority of MUSE.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 48 of its 64 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

48 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding14 Mar
  2. 2024VideoMamba: State Space Model for Efficient Video Understanding11 Mar

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2017Dense-Captioning Events in Videoscited by 789
  10. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  11. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
34 more
  1. 2020Multi-modal Transformer for Video Retrievalcited by 395
  2. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  3. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  4. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  5. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  6. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  7. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  8. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  9. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  10. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  11. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  12. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  13. 2021CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learningcited by 69
  14. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcarecited by 62
  15. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  16. 2023Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferringcited by 57
  17. 2024Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstructioncited by 56
  18. 2021On Pursuit of Designing Multi-modal Transformer for Video Groundingcited by 54
  19. 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
  20. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  21. 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
  22. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  23. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  24. 2024Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrievalcited by 33
  25. 2022Unsupervised Pre-training for Temporal Action Localization Taskscited by 31
  26. 2023Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignmentcited by 31
  27. 2022Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representationscited by 19
  28. 2024RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adaptercited by 16
  29. 2022Correspondence Matters for Video Referring Expression Comprehensioncited by 10
  30. 2023Exploiting Auxiliary Caption for Video Groundingcited by 10
  31. 2023Improving Video Retrieval by Adaptive Margincited by 6
  32. 2024Uncertainty-aware sign language video retrieval with probability distribution modelingcited by 6
  33. 2023Improving Reference-based Distinctive Image Captioning with Contrastive Rewardscited by 4
  34. 2023Video Referring Expression Comprehension via Transformer with Content-conditioned Querycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2026Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrievalcited by 23
  2. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  3. 2025MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusioncited by 8
  4. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  5. 2025MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videoscited by 2
  6. 2026Understanding the Performance Plateau in Text-to-Video Retrieval: A Comprehensive Empirical and Linguistic Analysiscited by 0
  7. 2026DREAM: Extending Vision-Language Models with Dual-Objective Encoding for Cross-Modal Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge
  2. 2025MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval
  3. 2026TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval
  4. 2026MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?
  5. 2026VeRVE: Versatile Retrieval for Videos via Unified Embeddings
  6. 2024MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning
  7. 2025Mitigating Semantic Collapse in Partially Relevant Video Retrieval
  8. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models