mlsanityData to 14 Oct 2026

Encoding and Controlling Global Semantics for Long-form Video Question Answering

arXiv 2405.19723 ↗First posted 30 May 2024cs.CV cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence (C^3) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets. The code, model, and data have been made available at https://nguyentthong.github.io/Long_form_VideoQA.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

46 of 67 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  9. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  10. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  11. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  12. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
34 more
  1. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  2. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  3. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  4. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  5. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  6. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 154
  7. 2021Towards Long-Form Video Understandingcited by 146
  8. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  9. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 108
  10. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  11. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 89
  12. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  13. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  14. 2021Contrastive Learning for Neural Topic Modelcited by 48
  15. 2023Effective Neural Topic Modeling with Embedding Clustering Regularizationcited by 46
  16. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  17. 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 37
  18. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  19. 2024FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Modelcited by 35
  20. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  21. 2023InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modelingcited by 24
  22. 2021Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillationcited by 22
  23. 2021Learning Relation Alignment for Calibrated Cross-modal Retrievalcited by 21
  24. 2021Enriching and Controlling Global Semantics for Text Summarizationcited by 20
  25. 2024On the Affinity, Rationality, and Diversity of Hierarchical Topic Modelingcited by 18
  26. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignmentcited by 17
  27. 2024Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusioncited by 16
  28. 2022Adaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Predictionscited by 13
  29. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  30. 2023Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representationcited by 12
  31. 2023DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Groundingcited by 11
  32. 2023Gradient-Boosted Decision Tree for Listwise Context Model in Multimodal Review Helpfulness Predictioncited by 7
  33. 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignmentcited by 5
  34. 2023READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modelingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  2. 2024Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generationcited by 9
  3. 2025TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systemscited by 8
  4. 2026Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoderscited by 7
  5. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 7
  6. 2025PAVE: Patching and Adapting Video Large Language Modelscited by 4
  7. 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streamscited by 3
  8. 2025Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understandingcited by 1