# Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

arXiv: https://arxiv.org/abs/2408.07919 · HTML: https://mlsanity.com/paper/2408.07919/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2024.

## Built on by (most cited first, 14 of 14)

- [Gibberish is All You Need for Membership Inference Detection in Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2410.18371/) (2024)
- [Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries](https://mlsanity.com/paper/2507.16343/) (2025)
- [Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval](https://mlsanity.com/paper/2604.18360/) (2026)
- [ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology](https://mlsanity.com/paper/2511.02946/) (2025)
- [The Sound of Absence: Audio-Language Embedding Models Struggle with Negation](https://mlsanity.com/paper/2607.12290/) (2026)
- [Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping](https://mlsanity.com/paper/2505.13777/) (2025)
- [MoEScore: Mixture-of-Experts-Based Text-Audio Relevance Score Prediction for Text-to-Audio System Evaluation](https://mlsanity.com/paper/2601.06829/) (2026)
- [SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training](https://mlsanity.com/paper/2601.12594/) (2026)
- [WavLink: Compact Audio-Text Embeddings with a Global Whisper Token](https://mlsanity.com/paper/2601.15118/) (2026)
- [CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation](https://mlsanity.com/paper/2603.19615/) (2026)
- [Local Multimodal Music Alignment from Global Supervision](https://mlsanity.com/paper/2607.10023/) (2026)
- [Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception](https://mlsanity.com/paper/2609.15215/) (2026)
- [VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale](https://mlsanity.com/paper/2609.32016/) (2026)
- [CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulations](https://mlsanity.com/paper/2609.33433/) (2026)

## Abstract (arXiv)

Recent advances have been witnessed in audio-language joint learning, such as CLAP, that shows much success in multi-modal understanding tasks. These models usually aggregate uni-modal local representations, namely frame or word features, into global ones, on which the contrastive loss is employed to reach coarse-grained cross-modal alignment. However, frame-level correspondence with texts may be ignored, making it ill-posed on explainability and fine-grained challenges which may also undermine performances on coarse-grained tasks. In this work, we aim to improve both coarse- and fine-grained audio-language alignment in large-scale contrastive pre-training. To unify the granularity and latent distribution of two modalities, a shared codebook is adopted to represent multi-modal global features with common bases, and each codeword is regularized to encode modality-shared semantics, bridging the gap between frame and word features. Based on it, a locality-aware block is involved to purify local patterns, and a hard-negative guided loss is devised to boost alignment. Experiments on eleven zero-shot coarse- and fine-grained tasks suggest that our model not only surpasses the baseline CLAP significantly but also yields superior or competitive results compared to current SOTA works.