mlsanityData to 14 Oct 2026

FLAP: Fast Language-Audio Pre-training

arXiv 2311.01615 ↗First posted 2 Nov 2023cs.SD cs.CL eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

We propose Fast Language-Audio Pre-training (FLAP), a self-supervised approach that efficiently and effectively learns aligned audio and language representations through masking, contrastive learning and reconstruction. For efficiency, FLAP randomly drops audio spectrogram tokens, focusing solely on the remaining ones for self-supervision. Through inter-modal contrastive learning, FLAP learns to align paired audio and text representations in a shared latent space. Notably, FLAP leverages multiple augmented views via masking for inter-modal contrast and learns to reconstruct the masked portion of audio tokens. Moreover, FLAP leverages large language models (LLMs) to augment the text inputs, contributing to improved performance. These approaches lead to more robust and informative audio-text representations, enabling FLAP to achieve state-of-the-art (SoTA) performance on audio-text retrieval tasks on AudioCaps (achieving 53.0% R@1) and Clotho (achieving 25.5% R@1).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
20 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  7. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  8. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  9. 2019Clotho: An Audio Captioning Datasetcited by 358
  10. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  11. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  12. 2022Masked Autoencoders that Listencited by 273
  13. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  14. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  15. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  16. 2022MAViL: Masked Audio-Video Learnerscited by 59
  17. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  18. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  19. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  20. 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  2. 2024Bridging Language Gaps in Audio-Text Retrievalcited by 6
  3. 2025FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusioncited by 6
  4. 2024Dissecting Temporal Understanding in Text-to-Audio Retrievalcited by 3
  5. 2026Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classificationcited by 1