# A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval

arXiv: https://arxiv.org/abs/2402.19106 · HTML: https://mlsanity.com/paper/2402.19106/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Feb 2024.

## Built on by (most cited first, 4 of 4)

- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Dissecting Temporal Understanding in Text-to-Audio Retrieval](https://mlsanity.com/paper/2409.00851/) (2024)
- [A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classification](https://mlsanity.com/paper/2409.13676/) (2024)
- [Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition](https://mlsanity.com/paper/2409.09611/) (2024)

## Abstract (arXiv)

Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, treating visual descriptions as audio descriptions is far from optimal. Even if audio class labels are present, they commonly are not very detailed, making them unsuited for text-audio retrieval. To exploit relevant audio information from video-text datasets, we introduce a methodology for generating audio-centric descriptions using Large Language Models (LLMs). In this work, we consider the egocentric video setting and propose three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset. Our approach for obtaining audio-centric descriptions gives significantly higher zero-shot performance than using the original visual-centric descriptions. Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset. Finally, we confirm that LLMs can be used to determine the difficulty of identifying the action associated with a sound.