mlsanityData to 7 Oct 2026

Audio Captioning Transformer

arXiv 2107.09817 ↗First posted 21 Jul 2021eess.AS cs.LG cs.SDcited by 53

Not on the map. Why?

Abstract (arXiv)

Audio captioning aims to automatically generate a natural language description of an audio clip. Most captioning models follow an encoder-decoder architecture, where the decoder predicts words based on the audio features extracted by the encoder. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are often used as the audio encoder. However, CNNs can be limited in modelling temporal relationships among the time frames in an audio signal, while RNNs can be limited in modelling the long-range dependencies among the time frames. In this paper, we propose an Audio Captioning Transformer (ACT), which is a full Transformer network based on an encoder-decoder architecture and is totally convolution-free. The proposed method has a better ability to model the global information within an audio signal as well as capture temporal relationships between audio events. We evaluate our model on AudioCaps, which is the largest audio captioning dataset publicly available. Our model shows competitive performance compared to other state-of-the-art approaches.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

18 of 27 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  9. 2019Clotho: An Audio Captioning Datasetcited by 354
  10. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  11. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  12. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
6 more
  1. 2021Investigating Local and Global Information for Automated Audio Captioning with Transfer Learningcited by 25
  2. 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
  3. 2019Audio Caption: Listen and Tellcited by 20
  4. 2019Weakly Labelled AudioSet Tagging with Attention Neural Networkscited by 17
  5. 2020Effects of Word-frequency based Pre- and Post- Processings for Audio Captioningcited by 12
  6. 2020WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Informationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 18; 2024: 9; 2025: 6; 2026: 3.

53 papers.

Sort:

By area:

  1. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  2. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  3. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  4. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  5. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  6. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 70
  7. 2022MAViL: Masked Audio-Video Learnerscited by 56
  8. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  9. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
  10. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  11. 2023Separate Anything You Describecited by 43
  12. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
38 more
  1. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  2. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 26
  3. 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 22
  4. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  5. 2022Leveraging Pre-trained BERT for Audio Captioningcited by 17
  6. 2023Fine-grained Audible Video Descriptioncited by 17
  7. 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
  8. 2021Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularizationcited by 12
  9. 2022Visually-Aware Audio Captioning With Adaptive Audio-Visual Attentioncited by 11
  10. 2022Exploring Train and Test-Time Augmentations for Audio-Language Learningcited by 11
  11. 2023Zero-shot audio captioning with audio-language model guidance and audio context keywordscited by 11
  12. 2024SoundingActions: Learning How Actions Sound from Narrated Egocentric Videoscited by 11
  13. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretrainingcited by 9
  14. 2023Weakly-supervised Automated Audio Captioning via text only trainingcited by 8
  15. 2025LAVCap: LLM-based Audio-Visual Captioning using Optimal Transportcited by 7
  16. 2024AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioningcited by 6
  17. 2022Local Information Assisted Attention-free Decoder for Audio Captioningcited by 5
  18. 2022Joint Speech Recognition and Audio Captioningcited by 5
  19. 2023Generating Realistic Images from In-the-wild Soundscited by 4
  20. 2022Towards Generating Diverse Audio Captions via Adversarial Trainingcited by 3
  21. 2023Zero-Shot Audio Captioning via Audibility Guidancecited by 3
  22. 2024Dissecting Temporal Understanding in Text-to-Audio Retrievalcited by 3
  23. 2024EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performancecited by 3
  24. 2025Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioningcited by 3
  25. 2022Automated Audio Captioning and Language-Based Audio Retrievalcited by 2
  26. 2022Automated Audio Captioning via Fusion of Low- and High- Dimensional Featurescited by 2
  27. 2023Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglementcited by 2
  28. 2024Towards Diverse and Efficient Audio Captioning via Diffusion Modelscited by 2
  29. 2025Multiple Choice Learning of Low-Rank Adapters for Language Modelingcited by 2
  30. 2022Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similaritycited by 1
  31. 2024Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioningcited by 1
  32. 2025Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoderscited by 1
  33. 2025Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioningcited by 1
  34. 2022Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learningcited by 0
  35. 2023Efficient Audio Captioning Transformer with Patchout and Text Guidancecited by 0
  36. 2023Improving Audio Caption Fluency with Automatic Error Correctioncited by 0
  37. 2023Audio Difference Learning for Audio Captioningcited by 0
  38. 2024Distance Sampling-based Paraphraser Leveraging ChatGPT for Text Data Manipulationcited by 0

50 most cited of 53 All 53 (JSON)