# Investigating Local and Global Information for Automated Audio Captioning with Transfer Learning

arXiv: https://arxiv.org/abs/2102.11457 · HTML: https://mlsanity.com/paper/2102.11457/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2021.

## Built on by (most cited first, 26 of 26)

- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model](https://mlsanity.com/paper/2309.16058/) (2023)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [UnIVAL: Unified Model for Image, Video, Audio and Language Tasks](https://mlsanity.com/paper/2307.16184/) (2023)
- [eP-ALM: Efficient Perceptual Augmentation of Language Models](https://mlsanity.com/paper/2303.11403/) (2023)
- [Can Audio Captions Be Evaluated with Image Caption Metrics?](https://mlsanity.com/paper/2110.04684/) (2021)
- [RECAP: Retrieval-Augmented Audio Captioning](https://mlsanity.com/paper/2309.09836/) (2023)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization](https://mlsanity.com/paper/2108.04692/) (2021)
- [Zero-shot audio captioning with audio-language model guidance and audio context keywords](https://mlsanity.com/paper/2311.08396/) (2023)
- [Weakly-supervised Automated Audio Captioning via text only training](https://mlsanity.com/paper/2309.12242/) (2023)
- [Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning](https://mlsanity.com/paper/2110.07410/) (2021)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [Caption Feature Space Regularization for Audio Captioning](https://mlsanity.com/paper/2204.08409/) (2022)
- [Efficient Audio Captioning with Encoder-Level Knowledge Distillation](https://mlsanity.com/paper/2407.14329/) (2024)
- [Contrasting Deep Learning Models for Direct Respiratory Insufficiency Detection Versus Blood Oxygen Saturation Estimation](https://mlsanity.com/paper/2407.20989/) (2024)
- [Automatic Audio Captioning using Attention weighted Event based Embeddings](https://mlsanity.com/paper/2201.12352/) (2022)
- [A Temporal-oriented Broadcast ResNet for COVID-19 Detection](https://mlsanity.com/paper/2203.17012/) (2022)
- [Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learning](https://mlsanity.com/paper/2206.01918/) (2022)
- [An investigation on selecting audio pre-trained models for audio captioning](https://mlsanity.com/paper/2208.06127/) (2022)
- [Improving Audio Caption Fluency with Automatic Error Correction](https://mlsanity.com/paper/2306.10090/) (2023)

## Abstract (arXiv)

Automated audio captioning (AAC) aims at generating summarizing descriptions for audio clips. Multitudinous concepts are described in an audio caption, ranging from local information such as sound events to global information like acoustic scenery. Currently, the mainstream paradigm for AAC is the end-to-end encoder-decoder architecture, expecting the encoder to learn all levels of concepts embedded in the audio automatically. This paper first proposes a topic model for audio descriptions, comprehensively analyzing the hierarchical audio topics that are commonly covered. We then explore a transfer learning scheme to access local and global information. Two source tasks are identified to respectively represent local and global information, being Audio Tagging (AT) and Acoustic Scene Classification (ASC). Experiments are conducted on the AAC benchmark dataset Clotho and Audiocaps, amounting to a vast increase in all eight metrics with topic transfer learning. Further, it is discovered that local information and abstract representation learning are more crucial to AAC than global information and temporal relationship learning.