mlsanityData to 7 Oct 2026

Can I Trust Your Answer? Visually Grounded Video Question Answering

arXiv 2309.01327 ↗First posted 4 Sep 2023cs.CV cs.AI cs.MMcited by 133

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study visually grounded VideoQA in response to the emerging trends of utilizing pretraining techniques for video-language understanding. Specifically, by forcing vision-language models (VLMs) to answer questions and simultaneously provide visual evidence, we seek to ascertain the extent to which the predictions of such techniques are genuinely anchored in relevant video content, versus spurious correlations from language or irrelevant visual context. Towards this, we construct NExT-GQA -- an extension of NExT-QA with 10.5K temporal grounding (or location) labels tied to the original QA pairs. With NExT-GQA, we scrutinize a series of state-of-the-art VLMs. Through post-hoc attention analysis, we find that these models are extremely weak in substantiating the answers despite their strong QA performance. This exposes the limitation of current VLMs in making reliable predictions. As a remedy, we further explore and propose a grounded-QA method via Gaussian mask optimization and cross-modal learning. Experiments with different backbones demonstrate that this grounding mechanism improves both grounding and QA. With these efforts, we aim to push towards trustworthy VLMs in VQA systems. Our dataset and code are available at https://github.com/doc-doc/NExT-GQA.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 63 of its 79 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

63 of 79 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2021Video Swin Transformercited by 1,090
  9. 2021Scaling Vision Transformerscited by 993
  10. 2017Dense-Captioning Events in Videoscited by 789
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  13. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  14. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  15. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  16. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  17. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  18. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  19. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  20. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  21. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  22. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  23. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  24. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  25. 2022Learning Video Representations from Large Language Modelscited by 204
  26. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  27. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  28. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  29. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  30. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  31. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  32. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 116
  33. 2019Weakly Supervised Video Moment Retrieval From Text Queriescited by 85
  34. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  35. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  36. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  37. 2019Weakly-Supervised Video Moment Retrieval via Semantic Completion Networkcited by 69
  38. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 27; 2025: 64; 2026: 40.

133 papers.

Sort:

By area:

  1. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  2. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  3. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  4. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  5. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  6. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  7. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  8. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  9. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  10. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  11. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  12. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
38 more
  1. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  2. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  3. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  4. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  5. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  6. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  7. 2024Language Repository for Long Video Understandingcited by 55
  8. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  9. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  10. 2024Abductive Ego-View Accident Video Understanding for Safe Driving Perceptioncited by 43
  11. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  12. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  13. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  14. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  15. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  16. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  17. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  18. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 26
  19. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  20. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  21. 2025Human Motion Video Generation: A Surveycited by 25
  22. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 24
  23. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23
  24. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 23
  25. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  26. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  27. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 19
  28. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  29. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  30. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  31. 2024Chrono: A Simple Blueprint for Representing Time in MLLMscited by 17
  32. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
  33. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answeringcited by 12
  34. 2025Cross-modal Causal Relation Alignment for Video Question Groundingcited by 12
  35. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  36. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  37. 2025Visual Jenga: Discovering Object Dependencies via Counterfactual Inpaintingcited by 11
  38. 2025MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Groundingcited by 11

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering
  2. 2024VQA2: Visual Question Answering for Video Quality Assessment
  3. 2026Watch Before You Answer: Learning from Visually Grounded Post-Training
  4. 2024Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
  5. 2025Towards Understanding Visual Grounding in Visual Language Models
  6. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  7. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  8. 2022Guiding Visual Question Answering with Attention Priors