GAIS: Frame-Level Gated Audio-Visual Integration with Semantic Variance-Scaled Perturbation for Text-Video Retrieval
Not on the map. Why?
Abstract (arXiv)
Text-to-video retrieval requires precise alignment between language and temporally rich audio-video signals. However, existing methods often emphasize visual cues while underutilizing audio semantics or relying on coarse fusion strategies, resulting in suboptimal multimodal representations. We introduce GAIS, a retrieval framework that strengthens multimodal alignment from both representation and regularization perspectives. First, a Frame-level Gated Fusion (FGF) module adaptively integrates audio-visual features under textual guidance, enabling fine-grained temporal selection of informative frames. Second, a Semantic Variance-Scaled Perturbation (SVSP) mechanism regularizes the text embedding space by controlling perturbation magnitude in a semantics-aware manner. These two modules are complementary: FGF minimizes modality gaps through selective fusion, while SVSP improves embedding stability and discrimination. Extensive experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX demonstrate that GAIS consistently outperforms strong baselines across multiple retrieval metrics while maintaining notable computational efficiency.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 39 references that are in the map. First posted on arXiv 3 Aug 2025.
Builds on
36 of 39 references.
References · 36
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
- 2017Localizing Moments in Video with Natural Languagecited by 663
- 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
- 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
- 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
- 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
- 2020Multi-modal Transformer for Video Retrievalcited by 395
- 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
- 2020Self-Supervised MultiModal Versatile Networkscited by 276
24 more
- 2015A Dataset for Movie Descriptioncited by 195
- 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
- 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
- 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
- 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 128
- 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
- 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
- 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
- 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
- 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
- 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
- 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
- 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
- 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
- 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
- 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
- 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
- 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
- 2024Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrievalcited by 33
- 2021HANet: Hierarchical Alignment Networks for Video-Text Retrievalcited by 31
- 2023Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignmentcited by 21
- 2022Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representationscited by 19
- 2025Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrievalcited by 8
- 2025Expertized Caption Auto-Enhancement for Video-Text Retrievalcited by 4
Built on by
Not cited yet.