# LLM-AD: Large Language Model based Audio Description System

arXiv: https://arxiv.org/abs/2405.00983 · HTML: https://mlsanity.com/paper/2405.00983/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 15 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 13 of 13)

- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions](https://mlsanity.com/paper/2112.00431/) (2021)
- [TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking](https://mlsanity.com/paper/2104.00194/) (2021)
- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)

## Built on by (most cited first, 15 of 15)

- [AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description](https://mlsanity.com/paper/2407.15850/) (2024)
- [Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation](https://mlsanity.com/paper/2504.01020/) (2025)
- [DistinctAD: Distinctive Audio Description Generation in Contexts](https://mlsanity.com/paper/2411.18180/) (2024)
- [What You See is What You Ask: Evaluating Audio Descriptions](https://mlsanity.com/paper/2510.00808/) (2025)
- [Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies](https://mlsanity.com/paper/2410.08860/) (2024)
- [More than a Moment: Towards Coherent Sequences of Audio Descriptions](https://mlsanity.com/paper/2510.25440/) (2025)
- [Character-Centric Understanding of Animated Movies](https://mlsanity.com/paper/2509.12204/) (2025)
- [MCAD: Multimodal Context-Aware Audio Description Generation For Soccer](https://mlsanity.com/paper/2511.09448/) (2025)
- [REFRAMED: Towards Realistic Audio Description Generation for Movies](https://mlsanity.com/paper/2608.09765/) (2026)
- [Integrating Video and Text: A Balanced Approach to Multimodal Summary Generation and Evaluation](https://mlsanity.com/paper/2505.06594/) (2025)
- [How Do Novice Describers Edit AI Drafts? The Role of Draft Quality in Audio Description Authoring](https://mlsanity.com/paper/2605.05348/) (2026)
- [READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations](https://mlsanity.com/paper/2606.22766/) (2026)
- [Andha-Dhun: A First Look at Audio Descriptions in Hindi](https://mlsanity.com/paper/2607.06457/) (2026)
- [StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos](https://mlsanity.com/paper/2608.12549/) (2026)
- [AD4EDU: Benchmarking University Lecture Audio Description across Accessibility Standards, Metrics, and Learner Preferences](https://mlsanity.com/paper/2610.17042/) (2026)

## Abstract (arXiv)

The development of Audio Description (AD) has been a pivotal step forward in making video content more accessible and inclusive. Traditionally, AD production has demanded a considerable amount of skilled labor, while existing automated approaches still necessitate extensive training to integrate multimodal inputs and tailor the output from a captioning style to an AD style. In this paper, we introduce an automated AD generation pipeline that harnesses the potent multimodal and instruction-following capacities of GPT-4V(ision). Notably, our methodology employs readily available components, eliminating the need for additional training. It produces ADs that not only comply with established natural language AD production standards but also maintain contextually consistent character information across frames, courtesy of a tracking-based character recognition module. A thorough analysis on the MAD dataset reveals that our approach achieves a performance on par with learning-based methods in automated AD production, as substantiated by a CIDEr score of 20.5.