mlsanityData to 7 Oct 2026

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

arXiv 2312.09736 ↗First posted 15 Dec 2023cs.CL cs.SD eess.AScited by 13

Not on the map. Why?

Abstract (arXiv)

Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems to improve the quality of their responses, existing systems are competent only to incorporate the information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems' ignoring audio data as a deaf response. To overcome the deaf response problem, Hearing Enhanced Audio Response (HEAR) framework is proposed to perform sensible listening by selectively attending to audio whenever the question requires it. The HEAR framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. HEAR is validated on VGD datasets (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows effectiveness with various VGD systems.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

26 of 43 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2016Layer Normalizationcited by 5,579
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  12. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
14 more
  1. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  2. 2019Audio-Visual Scene-Aware Dialogcited by 122
  3. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  4. 2018End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Featurescited by 37
  5. 2022Selective Query-guided Debiasing for Video Corpus Moment Retrievalcited by 18
  6. 2022Information-Theoretic Text Hallucination Reduction for Video-grounded Dialoguecited by 18
  7. 2021Structured Co-reference Graph Attention for Video-grounded Dialoguecited by 17
  8. 2023SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrievalcited by 13
  9. 2020Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue Systemcited by 11
  10. 2021Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialoguescited by 11
  11. 2021VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Taskscited by 11
  12. 2022Video Dialog as Conversation about Objects Living in Space-Timecited by 11
  13. 2020DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generatorcited by 9
  14. 2019Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 5; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  2. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  3. 2024FRAG: Frequency Adapting Group for Diffusion Video Editingcited by 15
  4. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 15
  5. 2024TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animationcited by 12
  6. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8
  7. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 5
  8. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  9. 2024M2K-VDG: Model-Adaptive Multimodal Knowledge Anchor Enhanced Video-grounded Dialogue Generationcited by 3
  10. 2025Occlusion-robust Stylization for Drawing-based 3D Animationcited by 3
  11. 2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoningcited by 3
  12. 2025Semantic Compression via Multimodal Representation Learningcited by 0
1 more
  1. 2026Hypergraph-Regularized Gramian Volumes for Multimodal Retrievalcited by 0