mlsanityData to 7 Oct 2026

Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization

arXiv 2108.04692 ↗First posted 10 Aug 2021cs.CL cs.SD eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we examine the use of Transfer Learning using Pretrained Audio Neural Networks (PANNs), and propose an architecture that is able to better leverage the acoustic features provided by PANNs for the Automated Audio Captioning Task. We also introduce a novel self-supervised objective, Reconstruction Latent Space Similarity Regularization (RLSSR). The RLSSR module supplements the training of the model by minimizing the similarity between the encoder and decoder embedding. The combination of both methods allows us to surpass state of the art results by a significant margin on the Clotho dataset across several metrics and benchmarks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

17 of 23 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2015Fast R-CNNcited by 2,930
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  10. 2019Clotho: An Audio Captioning Datasetcited by 354
  11. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  12. 2021Audio Captioning Transformercited by 53
5 more
  1. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  2. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  3. 2021Investigating Local and Global Information for Automated Audio Captioning with Transfer Learningcited by 25
  4. 2020Audio Captioning using Gated Recurrent Unitscited by 4
  5. 2019Duration robust weakly supervised sound event detectioncited by 2

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 4; 2024: 1; 2025: 1.

12 papers.

  1. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  2. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  3. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 26
  4. 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 22
  5. 2022Leveraging Pre-trained BERT for Audio Captioningcited by 17
  6. 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
  7. 2023Weakly-supervised Automated Audio Captioning via text only trainingcited by 8
  8. 2024AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioningcited by 6
  9. 2022Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similaritycited by 1
  10. 2022Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learningcited by 0
  11. 2022Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Losscited by 0
  12. 2025Comprehensive Evaluation of CNN-Based Audio Tagging Models on Resource-Constrained Devicescited by 0