# T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

arXiv: https://arxiv.org/abs/2404.17806 · HTML: https://mlsanity.com/paper/2404.17806/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [Audio Retrieval with WavText5K and CLAP Training](https://mlsanity.com/paper/2209.14275/) (2022)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [Audio-Text Models Do Not Yet Leverage Natural Language](https://mlsanity.com/paper/2303.10667/) (2023)
- [Retrieval-Augmented Text-to-Audio Generation](https://mlsanity.com/paper/2309.08051/) (2023)
- [BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data](https://mlsanity.com/paper/2303.07902/) (2023)
- [Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Loss](https://mlsanity.com/paper/2303.05681/) (2023)
- [Enhance Temporal Relations in Audio Captioning with Sound Event Detection](https://mlsanity.com/paper/2306.01533/) (2023)

## Built on by (most cited first, 18 of 18)

- [Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound](https://mlsanity.com/paper/2408.11915/) (2024)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models](https://mlsanity.com/paper/2409.06223/) (2024)
- [CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation](https://mlsanity.com/paper/2410.02271/) (2024)
- [Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning](https://mlsanity.com/paper/2501.14680/) (2025)
- [GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models](https://mlsanity.com/paper/2506.17886/) (2025)
- [SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization](https://mlsanity.com/paper/2509.21033/) (2025)
- [Spatial Audio Question Answering and Reasoning on Dynamic Source Movements](https://mlsanity.com/paper/2602.16334/) (2026)
- [DreamAudio: Customized Text-to-Audio Generation with Diffusion Models](https://mlsanity.com/paper/2509.06027/) (2025)
- [Spatial Audio Motion Understanding and Reasoning](https://mlsanity.com/paper/2509.14666/) (2025)
- [The Interpretation Gap in Text-to-Music Generation Models](https://mlsanity.com/paper/2407.10328/) (2024)
- [Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairs](https://mlsanity.com/paper/2408.09269/) (2024)
- [AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?](https://mlsanity.com/paper/2509.17641/) (2025)
- [Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings](https://mlsanity.com/paper/2607.03806/) (2026)
- [The Sound of Absence: Audio-Language Embedding Models Struggle with Negation](https://mlsanity.com/paper/2607.12290/) (2026)
- [FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation](https://mlsanity.com/paper/2608.22111/) (2026)
- [Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings](https://mlsanity.com/paper/2601.11262/) (2026)
- [CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulations](https://mlsanity.com/paper/2609.33433/) (2026)

## Abstract (arXiv)

Contrastive language-audio pretraining~(CLAP) has been developed to align the representations of audio and language, achieving remarkable performance in retrieval and classification tasks. However, current CLAP struggles to capture temporal information within audio and text features, presenting substantial limitations for tasks such as audio retrieval and generation. To address this gap, we introduce T-CLAP, a temporal-enhanced CLAP model. We use Large Language Models~(LLMs) and mixed-up strategies to generate temporal-contrastive captions for audio clips from extensive audio-text datasets. Subsequently, a new temporal-focused contrastive loss is designed to fine-tune the CLAP model by incorporating these synthetic data. We conduct comprehensive experiments and analysis in multiple downstream tasks. T-CLAP shows improved capability in capturing the temporal relationship of sound events and outperforms state-of-the-art models by a significant margin.