mlsanityData to 14 Oct 2026

Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies

arXiv 2410.08860 ↗First posted 11 Oct 2024cs.CL cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Audio descriptions (ADs) function as acoustic commentaries designed to assist blind persons and persons with visual impairments in accessing digital media content on television and in movies, among other settings. As an accessibility service typically provided by trained AD professionals, the generation of ADs demands significant human effort, making the process both time-consuming and costly. Recent advancements in natural language processing (NLP) and computer vision (CV), particularly in large language models (LLMs) and vision-language models (VLMs), have allowed for getting a step closer to automatic AD generation. This paper reviews the technologies pertinent to AD generation in the era of LLMs and VLMs: we discuss how state-of-the-art NLP and CV technologies can be applied to generate ADs and identify essential research directions for the future.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 89 of its 167 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

89 of 167 references.

References · 89

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  13. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  14. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  15. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  16. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  17. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  18. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  19. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  20. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  21. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  22. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  23. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  24. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  25. 2016Movie Descriptioncited by 181
  26. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  27. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  28. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 115
  29. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
  30. 2020Captioning Images Taken by People Who Are Blindcited by 110
  31. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 108
  32. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  33. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  34. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  35. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 68
  36. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  37. 2021Lite Vision Transformer with Enhanced Self-Attentioncited by 64
  38. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 56

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025What You See is What You Ask: Evaluating Audio Descriptionscited by 8
  2. 2026REFRAMED: Towards Realistic Audio Description Generation for Moviescited by 1
  3. 2026Making Videos Accessible for Blind and Low Vision Users Using a Multimodal Agent Video Playercited by 0
  4. 2026Andha-Dhun: A First Look at Audio Descriptions in Hindicited by 0
  5. 2026AD4EDU: Benchmarking University Lecture Audio Description across Accessibility Standards, Metrics, and Learner Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ADx3: A Collaborative Workflow for High-Quality Accessible Audio Description ↗
  2. 2025DescribePro: Collaborative Audio Description with Human-AI Interaction ↗
  3. 2026ADCanvas: Accessible and Conversational Audio Description Authoring for Blind and Low Vision Creators
  4. 2026ViDscribe: Multimodal AI for Customizing Audio Description and Question Answering in Online Videos
  5. 2026READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations
  6. 2026Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters
  7. 2026StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos
  8. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation