# Cacophony: An Improved Contrastive Audio-Text Model

arXiv: https://arxiv.org/abs/2402.06986 · HTML: https://mlsanity.com/paper/2402.06986/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2024.

## Built on by (most cited first, 18 of 18)

- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [Bridging Language Gaps in Audio-Text Retrieval](https://mlsanity.com/paper/2406.07012/) (2024)
- [CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation](https://mlsanity.com/paper/2410.02271/) (2024)
- [Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2511.16757/) (2025)
- [GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models](https://mlsanity.com/paper/2506.17886/) (2025)
- [SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization](https://mlsanity.com/paper/2509.21033/) (2025)
- [Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio](https://mlsanity.com/paper/2607.18666/) (2026)
- [Fusing Audio and Metadata Embeddings Improves Language-based Audio Retrieval](https://mlsanity.com/paper/2406.15897/) (2024)
- [Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval](https://mlsanity.com/paper/2408.11641/) (2024)
- [Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders](https://mlsanity.com/paper/2501.03038/) (2025)
- [Sample-efficient Integration of New Modalities into Large Language Models](https://mlsanity.com/paper/2509.04606/) (2025)
- [Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods](https://mlsanity.com/paper/2603.25767/) (2026)
- [Testing chatbots on the creation of encoders for audio conditioned image generation](https://mlsanity.com/paper/2509.09717/) (2025)
- [SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training](https://mlsanity.com/paper/2601.12594/) (2026)
- [Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization](https://mlsanity.com/paper/2604.16247/) (2026)
- [VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale](https://mlsanity.com/paper/2609.32016/) (2026)

## Abstract (arXiv)

Despite recent advancements, audio-text models still lag behind their image-text counterparts in scale and performance. In this paper, we propose to improve both the data scale and the training procedure of audio-text contrastive models. Specifically, we craft a large-scale audio-text dataset containing 13,000 hours of text-labeled audio, using pretrained language models to process noisy text descriptions and automatic captioning to obtain text descriptions for unlabeled audio samples. We first train on audio-only data with a masked autoencoder (MAE) objective, which allows us to benefit from the scalability of unlabeled audio datasets. We then train a contrastive model with an auxiliary captioning objective with the audio encoder initialized from the MAE model. Our final model, which we name Cacophony, achieves state-of-the-art performance on audio-text retrieval tasks, and exhibits competitive results on the HEAR benchmark and other downstream tasks such as zero-shot classification.