# Local Information Assisted Attention-free Decoder for Audio Captioning

arXiv: https://arxiv.org/abs/2201.03217 · HTML: https://mlsanity.com/paper/2201.03217/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 10 Jan 2022.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [An Attention Free Transformer](https://mlsanity.com/paper/2105.14103/) (2021)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [A Transformer-based Audio Captioning Model with Keyword Estimation](https://mlsanity.com/paper/2007.00222/) (2020)
- [An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning](https://mlsanity.com/paper/2108.02752/) (2021)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Diverse Audio Captioning via Adversarial Training](https://mlsanity.com/paper/2110.06691/) (2021)

## Built on by (most cited first, 5 of 5)

- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning](https://mlsanity.com/paper/2407.07801/) (2024)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning](https://mlsanity.com/paper/2502.05435/) (2025)

## Abstract (arXiv)

Automated audio captioning aims to describe audio data with captions using natural language. Existing methods often employ an encoder-decoder structure, where the attention-based decoder (e.g., Transformer decoder) is widely used and achieves state-of-the-art performance. Although this method effectively captures global information within audio data via the self-attention mechanism, it may ignore the event with short time duration, due to its limitation in capturing local information in an audio signal, leading to inaccurate prediction of captions. To address this issue, we propose a method using the pretrained audio neural networks (PANNs) as the encoder and local information assisted attention-free Transformer (LocalAFT) as the decoder. The novelty of our method is in the proposal of the LocalAFT decoder, which allows local information within an audio signal to be captured while retaining the global information. This enables the events of different duration, including short duration, to be captured for more precise caption generation. Experiments show that our method outperforms the state-of-the-art methods in Task 6 of the DCASE 2021 Challenge with the standard attention-based decoder for caption generation.