mlsanityData to 14 Oct 2026

Self-Chained Image-Language Model for Video Localization and Question Answering

arXiv 2305.06988 ↗First posted 11 May 2023cs.CV cs.AI cs.CLcited by 203

Not on the map. Why?

Cited by 64 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent studies have shown promising results on utilizing large pre-trained image-language models for video question answering. While these image-language models can efficiently bootstrap the representation learning of video-language models, they typically concatenate uniformly sampled video frames as visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant to the language query, such uniform frame sampling can often lead to missing important visual cues. Although humans often find a video moment to focus on and rewind the moment to answer questions, training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Self-Chained Video Localization-Answering (SeViLA), a novel framework that leverages a single image-language model (BLIP-2) to tackle both temporal keyframe localization and QA on videos. SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2. We propose two ways of chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the Localizer finds multiple language-aware keyframes in a video, which the Answerer uses to predict the answer. Second, in the reverse chain, the Answerer generates keyframe pseudo-labels to refine the Localizer, alleviating the need for expensive video moment localization annotations. Our SeViLA framework outperforms several strong baselines on 5 challenging video QA and event prediction benchmarks, and achieves the state-of-the-art in both fine-tuning (NExT-QA, STAR) and zero-shot (NExT-QA, STAR, How2QA, VLEP) settings. We also analyze the impact of Localizer, comparisons of Localizer with other temporal localization models, pre-training/self-refinement of Localizer, and varying the number of keyframes.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). Builds on 80 of its 98 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

80 of 98 references.

References · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  13. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  14. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  16. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  17. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  18. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  19. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  20. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  21. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  22. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  23. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  24. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  25. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  26. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  27. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  28. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  29. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  30. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  31. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  32. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  33. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  34. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  35. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  36. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  37. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  38. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 64; 2025: 80; 2026: 42.

203 papers.

Sort:

By area:

  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  2. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  3. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  5. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  6. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  7. 2023Video Understanding with Large Language Models: A Surveycited by 206
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  10. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  11. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  12. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
38 more
  1. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  2. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  3. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  4. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  5. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  6. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  7. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  8. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  9. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 86
  10. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 83
  11. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  12. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  13. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  14. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  15. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69
  16. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  17. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 67
  18. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  19. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  20. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  21. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
  22. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  23. 2024VCA: Video Curious Agent for Long Video Understandingcited by 56
  24. 2024Language Repository for Long Video Understandingcited by 55
  25. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  26. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  27. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  28. 2024Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomalycited by 45
  29. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  30. 2024Abductive Ego-View Accident Video Understanding for Safe Driving Perceptioncited by 43
  31. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  32. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 43
  33. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  34. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 38
  35. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 37
  36. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
  37. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  38. 2023Correlation-Guided Query-Dependency Calibration for Video Temporal Groundingcited by 31

50 most cited of 203 All 203 (JSON)