mlsanityData to 14 Oct 2026

Semi-Parametric Video-Grounded Text Generation

arXiv 2301.11507 ↗First posted 27 Jan 2023cs.CV cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Efficient video-language modeling should consider the computational cost because of a large, sometimes intractable, number of video frames. Parametric approaches such as the attention mechanism may not be ideal since its computational cost quadratically increases as the video length increases. Rather, previous studies have relied on offline feature extraction or frame sampling to represent the video efficiently, focusing on cross-modal modeling in short video clips. In this paper, we propose a semi-parametric video-grounded text generation model, SeViT, a novel perspective on scalable video-language modeling toward long untrimmed videos. Treating a video as an external data store, SeViT includes a non-parametric frame retriever to select a few query-relevant frames from the data store for a given query and a parametric generator to effectively aggregate the frames with the query via late fusion methods. Experimental results demonstrate our method has a significant advantage in longer videos and causal video understanding. Moreover, our model achieves the new state of the art on four video-language datasets, iVQA (+4.8), Next-QA (+6.9), and Activitynet-QA (+4.8) in accuracy, and MSRVTT-Caption (+3.6) in CIDEr.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 56 of its 72 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

56 of 72 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2017Billion-scale similarity search with GPUscited by 2,497
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2021ViViT: A Video Vision Transformercited by 1,620
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  10. 2021Video Swin Transformercited by 1,091
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  13. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  14. 2017Dense-Captioning Events in Videoscited by 793
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  16. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  17. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  18. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  20. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  21. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  22. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  23. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  24. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  25. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  26. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  27. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  28. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  29. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  30. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  31. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  32. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  33. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  34. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  35. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  36. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  37. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 107
  38. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 3.

13 papers.

Sort:

By area:

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  2. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  3. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  4. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  5. 2024Language Repository for Long Video Understandingcited by 55
  6. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  7. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  8. 2024Enhancing Temporal Modeling of Video LLMs via Time Gatingcited by 10
  9. 2023Long Video Understanding with Learnable Retrieval in Video-Language Modelscited by 9
  10. 2024Glance and Focus: Memory Prompting for Multi-Event Video Question Answeringcited by 8
  11. 2024Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answeringcited by 5
  12. 2025AirVista-II: An Agentic System for Embodied UAVs Toward Dynamic Scene Semantic Understandingcited by 4
1 more
  1. 2025Perceive, Verify and Understand Long Video: Multi-Granular Perception and Active Verification via Interactive Agentscited by 2