mlsanityData to 7 Oct 2026

Empowering the Deaf and Hard of Hearing Community: Enhancing Video Captions Using Large Language Models

arXiv 2412.00342 ↗First posted 30 Nov 2024cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In today's digital age, video content is prevalent, serving as a primary source of information, education, and entertainment. However, the Deaf and Hard of Hearing (DHH) community often faces significant challenges in accessing video content due to the inadequacy of automatic speech recognition (ASR) systems in providing accurate and reliable captions. This paper addresses the urgent need to improve video caption quality by leveraging Large Language Models (LLMs). We present a comprehensive study that explores the integration of LLMs to enhance the accuracy and context-awareness of captions generated by ASR systems. Our methodology involves a novel pipeline that corrects ASR-generated captions using advanced LLMs. It explicitly focuses on models like GPT-3.5 and Llama2-13B due to their robust performance in language comprehension and generation tasks. We introduce a dataset representative of real-world challenges the DHH community faces to evaluate our proposed pipeline. Our results indicate that LLM-enhanced captions significantly improve accuracy, as evidenced by a notably lower Word Error Rate (WER) achieved by ChatGPT-3.5 (WER: 9.75%) compared to the original ASR captions (WER: 23.07%), ChatGPT-3.5 shows an approximate 57.72% improvement in WER compared to the original ASR captions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

20 of 43 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2019How Can We Know What Language Models Know?cited by 953
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  9. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  10. 2015Listen, Attend and Spellcited by 542
  11. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  12. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
8 more
  1. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  2. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  3. 2021Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASRcited by 40
  4. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 38
  5. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  6. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  7. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19
  8. 2017Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearingcited by 11

Built on by

Not cited yet.