# WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information

arXiv: https://arxiv.org/abs/2010.11098 · HTML: https://mlsanity.com/paper/2010.11098/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning](https://mlsanity.com/paper/2110.07410/) (2021)
- [Continual Learning for Automated Audio Captioning Using The Learning Without Forgetting Approach](https://mlsanity.com/paper/2107.08028/) (2021)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)

## Abstract (arXiv)

Automated audio captioning (AAC) is a novel task, where a method takes as an input an audio sample and outputs a textual description (i.e. a caption) of its contents. Most AAC methods are adapted from from image captioning of machine translation fields. In this work we present a novel AAC novel method, explicitly focused on the exploitation of the temporal and time-frequency patterns in audio. We employ three learnable processes for audio encoding, two for extracting the local and temporal information, and one to merge the output of the previous two processes. To generate the caption, we employ the widely used Transformer decoder. We assess our method utilizing the freely available splits of Clotho dataset. Our results increase previously reported highest SPIDEr to 17.3, from 16.2.