# GAIS: Frame-Level Gated Audio-Visual Integration with Semantic Variance-Scaled Perturbation for Text-Video Retrieval

arXiv: https://arxiv.org/abs/2508.01711 · HTML: https://mlsanity.com/paper/2508.01711/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 39 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss](https://mlsanity.com/paper/2109.04290/) (2021)
- [TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval](https://mlsanity.com/paper/2207.07852/) (2022)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval](https://mlsanity.com/paper/2104.10054/) (2021)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [CenterCLIP: Token Clustering for Efficient Text-Video Retrieval](https://mlsanity.com/paper/2205.00823/) (2022)
- [Bridging Video-text Retrieval with Multiple Choice Questions](https://mlsanity.com/paper/2201.04850/) (2022)
- [Disentangled Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2203.07111/) (2022)
- [CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment](https://mlsanity.com/paper/2209.06430/) (2022)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [UATVR: Uncertainty-Adaptive Text-Video Retrieval](https://mlsanity.com/paper/2301.06309/) (2023)
- [DiffusionRet: Generative Text-Video Retrieval with Diffusion Model](https://mlsanity.com/paper/2303.09867/) (2023)
- [ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound](https://mlsanity.com/paper/2204.02874/) (2022)

## Abstract (arXiv)

Text-to-video retrieval requires precise alignment between language and temporally rich audio-video signals. However, existing methods often emphasize visual cues while underutilizing audio semantics or relying on coarse fusion strategies, resulting in suboptimal multimodal representations. We introduce GAIS, a retrieval framework that strengthens multimodal alignment from both representation and regularization perspectives. First, a Frame-level Gated Fusion (FGF) module adaptively integrates audio-visual features under textual guidance, enabling fine-grained temporal selection of informative frames. Second, a Semantic Variance-Scaled Perturbation (SVSP) mechanism regularizes the text embedding space by controlling perturbation magnitude in a semantics-aware manner. These two modules are complementary: FGF minimizes modality gaps through selective fusion, while SVSP improves embedding stability and discrimination. Extensive experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX demonstrate that GAIS consistently outperforms strong baselines across multiple retrieval metrics while maintaining notable computational efficiency.