mlsanityData to 14 Oct 2026

AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description

arXiv 2407.15850 ↗First posted 22 Jul 2024cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs), and develop visual and text prompting strategies for this task. Our contributions are three-fold: (i) We demonstrate that a VLM can successfully name and refer to characters if directly prompted with character information through visual indications without requiring any fine-tuning; (ii) A two-stage process is developed to generate ADs, with the first stage asking the VLM to comprehensively describe the video, followed by a second stage utilising a LLM to summarise dense textual information into one succinct AD sentence; (iii) A new dataset for TV audio description is formulated. Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

43 of 50 references.

References · 43

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2023Mistral 7Bcited by 3,164
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
31 more
  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  2. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  3. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  4. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  5. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  6. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  7. 2022Visual Prompting via Image Inpaintingcited by 250
  8. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  9. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  10. 2015A Dataset for Movie Descriptioncited by 195
  11. 2016Movie Descriptioncited by 181
  12. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  13. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  14. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 155
  15. 2016Video Summarization with Long Short-term Memorycited by 142
  16. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  17. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 108
  18. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  19. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  20. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  21. 2023AutoAD: Movie Description in Contextcited by 50
  22. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 49
  23. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  24. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  25. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  26. 2024LLM-AD: Large Language Model based Audio Description Systemcited by 15
  27. 2018Person Search in Videos with One Portrait Through Visual and Temporal Linkscited by 14
  28. 2024Contextual AD Narration with Interleaved Multimodal Sequencecited by 14
  29. 2024MICap: A Unified Model for Identity-aware Movie Descriptionscited by 9
  30. 2024Look, Listen and Recognise: Character-Aware Audio-Visual Subtitlingcited by 8
  31. 2023How you feelin'? Learning Emotions and Mental States in Movie Scenescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 12.

23 papers.

Sort:

By area:

  1. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  2. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  3. 2024DistinctAD: Distinctive Audio Description Generation in Contextscited by 12
  4. 2025What You See is What You Ask: Evaluating Audio Descriptionscited by 8
  5. 2024Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologiescited by 5
  6. 2025More than a Moment: Towards Coherent Sequences of Audio Descriptionscited by 5
  7. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  8. 2025Character-Centric Understanding of Animated Moviescited by 3
  9. 2025Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questionscited by 1
  10. 2025MCAD: Multimodal Context-Aware Audio Description Generation For Soccercited by 1
  11. 2026REFRAMED: Towards Realistic Audio Description Generation for Moviescited by 1
  12. 2026From Visual Cues to Spoken Narration: Rethinking Audio Descriptioncited by 1
11 more
  1. 2025Integrating Video and Text: A Balanced Approach to Multimodal Summary Generation and Evaluationcited by 0
  2. 2026How Do Novice Describers Edit AI Drafts? The Role of Draft Quality in Audio Description Authoringcited by 0
  3. 2026StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Datasetcited by 0
  4. 2026READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generationscited by 0
  5. 2026Andha-Dhun: A First Look at Audio Descriptions in Hindicited by 0
  6. 2026StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Descriptioncited by 0
  7. 2026StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videoscited by 0
  8. 2026TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Filmcited by 0
  9. 2026What, When, and How: Audio Description as Constrained Global Optimizationcited by 0
  10. 2026AD4EDU: Benchmarking University Lecture Audio Description across Accessibility Standards, Metrics, and Learner Preferencescited by 0
  11. 2026Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification
  2. 2025DescribePro: Collaborative Audio Description with Human-AI Interaction ↗
  3. 2024Movie101v2: Improved Movie Narration Benchmark
  4. 2024Character-aware audio-visual subtitling in context
  5. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation
  6. 2026FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes
  7. 2024Audio-visual training for improved grounding in video-text LLMs
  8. 2026SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks