mlsanityData to 14 Oct 2026

Large Language Models are Temporal and Causal Reasoners for Video Question Answering

arXiv 2310.15747 ↗First posted 24 Oct 2023cs.CVcited by 51

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. We observe that the LLMs provide effective priors in exploiting textitlinguistic shortcuts for temporal and causal reasoning in Video Question Answering (VideoQA). However, such priors often cause suboptimal results on VideoQA by leading the model to over-rely on questions, textiti.e., textitlinguistic bias, while ignoring visual content. This is also known as `ungrounded guesses' or `hallucinations'. To address this problem while leveraging LLMs' prior on VideoQA, we propose a novel framework, Flipped-VQA, encouraging the model to predict all the combinations of langleV, Q, Arangle triplet by flipping the source pair and the target label to understand their complex relationships, textiti.e., predict A, Q, and V given a VQ, VA, and QA pairs, respectively. In this paper, we develop LLaMA-VQA by applying Flipped-VQA to LLaMA, and it outperforms both LLMs-based and non-LLMs-based models on five challenging VideoQA benchmarks. Furthermore, our Flipped-VQA is a general framework that is applicable to various LLMs (OPT and GPT-J) and consistently improves their performances. We empirically demonstrate that Flipped-VQA not only enhances the exploitation of linguistic shortcuts but also mitigates the linguistic bias, which causes incorrect answers over-relying on the question. Code is available at https://github.com/mlvlab/Flipped-VQA.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 74 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

47 of 74 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
35 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  3. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  4. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  5. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  6. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  8. 2022Large Language Models Are Reasoning Teacherscited by 329
  9. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  10. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  11. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  12. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  13. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  14. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  15. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  16. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 185
  17. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  18. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  19. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  20. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  21. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  22. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 105
  23. 2023Mitigating Label Biases for In-context Learningcited by 89
  24. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  25. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  26. 2023Can large language models build causal graphs?cited by 73
  27. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  28. 2020What is More Likely to Happen Next? Video-and-Language Future Event Predictioncited by 62
  29. 2023Understanding Causality with Large Language Models: Feasibility and Opportunitiescited by 53
  30. 2020DramaQA: Character-Centered Video Story Understanding with Hierarchical QAcited by 45
  31. 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
  32. 2020Self-supervised pre-training and contrastive representation learning for multiple-choice video QAcited by 28
  33. 2020Hierarchical Conditional Relation Networks for Multimodal Video Question Answeringcited by 13
  34. 2022Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanationscited by 7
  35. 2023Open-vocabulary Video Question Answering: A New Benchmark for Evaluating the Generalizability of Video Question Answering Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 20; 2025: 19; 2026: 8.

51 papers.

Sort:

By area:

  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  2. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  3. 2023Video Understanding with Large Language Models: A Surveycited by 206
  4. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  5. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  6. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  7. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  8. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  9. 2024Language Repository for Long Video Understandingcited by 55
  10. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  11. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  12. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
38 more
  1. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  2. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  3. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  4. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 21
  5. 2024A Causal Explainable Guardrails for Large Language Modelscited by 18
  6. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 18
  7. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  8. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  9. 2025Bisecle: Binding and Separation in Continual Learning for Video Language Understandingcited by 13
  10. 2024TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoningcited by 12
  11. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  12. 2024Question-Answering Dense Video Eventscited by 11
  13. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  14. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?cited by 10
  15. 2025How Can Objects Help Video-Language Understanding?cited by 8
  16. 2024Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignmentcited by 6
  17. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoderscited by 6
  18. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  19. 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
  20. 2025Moment Sampling in Video LLMs for Long-Form Video QAcited by 5
  21. 2025PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglementcited by 5
  22. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
  23. 2025Large Language Models for Crash Detection in Video: A Survey of Methods, Datasets, and Challengescited by 3
  24. 2024Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queriescited by 2
  25. 2025EQ-TAA: Equivariant Traffic Accident Anticipation via Diffusion-Based Accident Video Synthesiscited by 2
  26. 2025PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behaviorcited by 2
  27. 2026LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streamscited by 2
  28. 2025Two Causally Related Needles in a Video Haystackcited by 1
  29. 2025See What You Need: Query-Aware Visual Intelligence through Reasoning-Perception Loopscited by 1
  30. 2025Beyond Isolated Facts: Synthesizing Narrative and Grounded Supervision for VideoQAcited by 1
  31. 2026Don't Pause: Streaming Video-Language Synchrony for Online Video Understandingcited by 1
  32. 2026EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidencecited by 1
  33. 2026SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selectioncited by 1
  34. 2024Top-down Activity Representation Learning for Video Question Answeringcited by 0
  35. 2025REVEAL: Relation-based Video Representation Learning for Video-Question-Answeringcited by 0
  36. 2026Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answeringcited by 0
  37. 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
  38. 2026Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Modelscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LeAdQA: LLM-Driven Context-Aware Temporal Grounding for Video Question Answering
  2. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  3. 2025Team of One: Cracking Complex Video QA with Model Synergy
  4. 2025ReasVQA: Advancing VideoQA with Imperfect Reasoning Process
  5. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  6. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding
  7. 2022Invariant Grounding for Video Question Answering
  8. 2024Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering