# Do Audio-Language Models Understand Linguistic Variations?

arXiv: https://arxiv.org/abs/2410.16505 · HTML: https://mlsanity.com/paper/2410.16505/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [The Benefit Of Temporally-Strong Labels In Audio Event Classification](https://mlsanity.com/paper/2105.07031/) (2021)
- [InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval](https://mlsanity.com/paper/2301.01820/) (2023)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [Leveraging LLMs for Unsupervised Dense Retriever Ranking](https://mlsanity.com/paper/2402.04853/) (2024)
- [Fine-tuning CLIP Text Encoders with Two-step Paraphrasing](https://mlsanity.com/paper/2402.15120/) (2024)

## Built on by (most cited first, 4 of 4)

- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval](https://mlsanity.com/paper/2604.18360/) (2026)
- [SpurAudio: A Benchmark for Studying Shortcut Learning in Few-Shot Audio Classification](https://mlsanity.com/paper/2605.13672/) (2026)
- [CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulations](https://mlsanity.com/paper/2609.33433/) (2026)

## Abstract (arXiv)

Open-vocabulary audio language models (ALMs), like Contrastive Language Audio Pretraining (CLAP), represent a promising new paradigm for audio-text retrieval using natural language queries. In this paper, for the first time, we perform controlled experiments on various benchmarks to show that existing ALMs struggle to generalize to linguistic variations in textual queries. To address this issue, we propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations agnostic to linguistic variations. Specifically, we reformulate the contrastive loss used in CLAP architectures by introducing a multi-view contrastive learning objective, where paraphrases are treated as different views of the same audio scene and use this for training. Our proposed approach improves the text-to-audio retrieval performance of CLAP by 0.8%-13% across benchmarks and enhances robustness to linguistic variation.