# Enhance Temporal Relations in Audio Captioning with Sound Event Detection

arXiv: https://arxiv.org/abs/2306.01533 · HTML: https://mlsanity.com/paper/2306.01533/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2023.

## Built on by (most cited first, 4 of 4)

- [T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2404.17806/) (2024)
- [DiffSED: Sound Event Detection with Denoising Diffusion](https://mlsanity.com/paper/2308.07293/) (2023)
- [Efficient Audio Captioning with Encoder-Level Knowledge Distillation](https://mlsanity.com/paper/2407.14329/) (2024)
- [Audio Difference Learning for Audio Captioning](https://mlsanity.com/paper/2309.08141/) (2023)

## Abstract (arXiv)

Automated audio captioning aims at generating natural language descriptions for given audio clips, not only detecting and classifying sounds, but also summarizing the relationships between audio events. Recent research advances in audio captioning have introduced additional guidance to improve the accuracy of audio events in generated sentences. However, temporal relations between audio events have received little attention while revealing complex relations is a key component in summarizing audio content. Therefore, this paper aims to better capture temporal relationships in caption generation with sound event detection (SED), a task that locates events' timestamps. We investigate the best approach to integrate temporal information in a captioning model and propose a temporal tag system to transform the timestamps into comprehensible relations. Results evaluated by the proposed temporal metrics suggest that great improvement is achieved in terms of temporal relation generation.