mlsanityData to 7 Oct 2026

Do Audio-Language Models Understand Linguistic Variations?

arXiv 2410.16505 ↗First posted 21 Oct 2024cs.SD cs.LG eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Open-vocabulary audio language models (ALMs), like Contrastive Language Audio Pretraining (CLAP), represent a promising new paradigm for audio-text retrieval using natural language queries. In this paper, for the first time, we perform controlled experiments on various benchmarks to show that existing ALMs struggle to generalize to linguistic variations in textual queries. To address this issue, we propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations agnostic to linguistic variations. Specifically, we reformulate the contrastive loss used in CLAP architectures by introducing a multi-view contrastive learning objective, where paraphrases are treated as different views of the same audio scene and use this for training. Our proposed approach improves the text-to-audio retrieval performance of CLAP by 0.8%-13% across benchmarks and enhances robustness to linguistic variation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

16 of 28 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  4. 2019Clotho: An Audio Captioning Datasetcited by 354
  5. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  6. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  7. 2021Generating Datasets with Pretrained Language Modelscited by 192
  8. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  9. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  10. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  11. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  12. 2023Natural Language Supervision for General-Purpose Audio Representationscited by 60
4 more
  1. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  2. 2022On the Evaluation Metrics for Paraphrase Generationcited by 32
  3. 2024Leveraging LLMs for Unsupervised Dense Retriever Rankingcited by 19
  4. 2024Fine-tuning CLIP Text Encoders with Two-step Paraphrasingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  2. 2026Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrievalcited by 3
  3. 2026SpurAudio: A Benchmark for Studying Shortcut Learning in Few-Shot Audio Classificationcited by 2
  4. 2026CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulationscited by 0