mlsanityData to 14 Oct 2026

T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining

arXiv 2404.17806 ↗First posted 27 Apr 2024cs.SD cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Contrastive language-audio pretraining~(CLAP) has been developed to align the representations of audio and language, achieving remarkable performance in retrieval and classification tasks. However, current CLAP struggles to capture temporal information within audio and text features, presenting substantial limitations for tasks such as audio retrieval and generation. To address this gap, we introduce T-CLAP, a temporal-enhanced CLAP model. We use Large Language Models~(LLMs) and mixed-up strategies to generate temporal-contrastive captions for audio clips from extensive audio-text datasets. Subsequently, a new temporal-focused contrastive loss is designed to fine-tune the CLAP model by incorporating these synthetic data. We conduct comprehensive experiments and analysis in multiple downstream tasks. T-CLAP shows improved capability in capturing the temporal relationship of sound events and outperforms state-of-the-art models by a significant margin.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

29 of 35 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  10. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  11. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  12. 2019Clotho: An Audio Captioning Datasetcited by 358
17 more
  1. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  2. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  3. 2023Teaching CLIP to Count to Tencited by 198
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  5. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  6. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  7. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  8. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  9. 2021Audio Retrieval with Natural Language Queriescited by 49
  10. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  11. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 33
  12. 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30
  13. 2023Audio-Text Models Do Not Yet Leverage Natural Languagecited by 26
  14. 2023Retrieval-Augmented Text-to-Audio Generationcited by 24
  15. 2023BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Datacited by 15
  16. 2023Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Losscited by 12
  17. 2023Enhance Temporal Relations in Audio Captioning with Sound Event Detectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 6.

18 papers.

Sort:

By area:

  1. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  2. 2025GLAP: General contrastive audio-text pretraining across domains and languagescited by 10
  3. 2024Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Modelscited by 5
  4. 2024CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentationcited by 4
  5. 2025Diffusion based Text-to-Music Generation with Global and Local Text based Conditioningcited by 3
  6. 2025GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Modelscited by 3
  7. 2025SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularizationcited by 3
  8. 2026Spatial Audio Question Answering and Reasoning on Dynamic Source Movementscited by 3
  9. 2025DreamAudio: Customized Text-to-Audio Generation with Diffusion Modelscited by 2
  10. 2025Spatial Audio Motion Understanding and Reasoningcited by 2
  11. 2024The Interpretation Gap in Text-to-Music Generation Modelscited by 1
  12. 2024Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairscited by 1
6 more
  1. 2025AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?cited by 1
  2. 2026Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddingscited by 1
  3. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  4. 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
  5. 2026Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddingscited by 0
  6. 2026CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulationscited by 0