mlsanityData to 14 Oct 2026

Video Question Answering with Iterative Video-Text Co-Tokenization

arXiv 2208.00934 ↗First posted 1 Aug 2022cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Video question answering is a challenging task that requires understanding jointly the language input, the visual information in individual video frames, as well as the temporal information about the events occurring in the video. In this paper, we propose a novel multi-stream video encoder for video question answering that uses multiple video inputs and a new video-text iterative co-tokenization approach to answer a variety of questions related to videos. We experimentally evaluate the model on several datasets, such as MSRVTT-QA, MSVD-QA, IVQA, outperforming the previous state-of-the-art by large margins. Simultaneously, our model reduces the required GFLOPs from 150-360 to only 67, producing a highly efficient video question answering model.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 85 of its 101 references that are in the map. First posted on arXiv 1 Aug 2022.

Builds on

85 of 101 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2017The Kinetics Human Action Video Datasetcited by 1,909
  8. 2018SlowFast Networks for Video Recognitioncited by 1,867
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  10. 2021ViViT: A Video Vision Transformercited by 1,620
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
38 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  6. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  7. 2017Dense-Captioning Events in Videoscited by 793
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  10. 2017Localizing Moments in Video with Natural Languagecited by 663
  11. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  13. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  14. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  15. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  16. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  17. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  18. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  19. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  20. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  21. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  22. 2020Multi-modal Transformer for Video Retrievalcited by 395
  23. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  24. 2016Visual Dialogcited by 377
  25. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  26. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  27. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  28. 2017Temporal Relational Reasoning in Videoscited by 324
  29. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  30. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  31. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  32. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  33. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  34. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  35. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  36. 2015A Dataset for Movie Descriptioncited by 195
  37. 2016Movie Descriptioncited by 181
  38. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 3; 2025: 1; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  2. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  3. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  4. 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
  5. 2023MaMMUT: A Simple Architecture for Joint Learning for MultiModal Taskscited by 24
  6. 2025UniVid: The Open-Source Unified Video Modelcited by 19
  7. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  8. 2023TG-VQA: Ternary Game of Video Question Answeringcited by 6
  9. 2024Hierarchical Banzhaf Interaction for General Video-Language Representation Learningcited by 4
  10. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  11. 2022Compound Tokens: Channel Fusion for Vision-Language Representation Learningcited by 2
  12. 2023Harvest Video Foundation Models via Efficient Post-Pretrainingcited by 1
4 more
  1. 2024Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labelscited by 1
  2. 2024FILS: Self-Supervised Video Feature Prediction In Semantic Language Spacecited by 1
  3. 2023Joint Adaptive Representations for Image-Language Learningcited by 0
  4. 2023Diversifying Joint Vision-Language Tokenization Learningcited by 0