# End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Sampling

arXiv: https://arxiv.org/abs/2407.15047 · HTML: https://mlsanity.com/paper/2407.15047/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 21 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [All in One: Exploring Unified Video-Language Pre-training](https://mlsanity.com/paper/2203.07303/) (2022)
- [HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training](https://mlsanity.com/paper/2212.14546/) (2022)
- [Reparameterizable Subset Sampling via Continuous Relaxations](https://mlsanity.com/paper/1901.10517/) (2019)
- [MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering](https://mlsanity.com/paper/2010.14095/) (2020)
- [Self-supervised pre-training and contrastive representation learning for multiple-choice video QA](https://mlsanity.com/paper/2009.08043/) (2020)
- [Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge](https://mlsanity.com/paper/2402.16050/) (2024)
- [Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA](https://mlsanity.com/paper/2005.06409/) (2020)
- [Recent Advances in Video Question Answering: A Review of Datasets and Methods](https://mlsanity.com/paper/2101.05954/) (2021)
- [Fast and Robust Dynamic Hand Gesture Recognition via Key Frames Extraction and Feature Fusion](https://mlsanity.com/paper/1901.04622/) (2019)

## Built on by (most cited first, 19 of 19)

- [M-LLM Based Video Frame Selection for Efficient Video Understanding](https://mlsanity.com/paper/2502.19680/) (2025)
- [T*: Re-thinking Temporal Search for Long-Form Video Understanding](https://mlsanity.com/paper/2504.02259/) (2025)
- [StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding](https://mlsanity.com/paper/2508.15717/) (2025)
- [Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding](https://mlsanity.com/paper/2503.13139/) (2025)
- [Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models](https://mlsanity.com/paper/2503.16036/) (2025)
- [Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task](https://mlsanity.com/paper/2512.10359/) (2025)
- [ReasVQA: Advancing VideoQA with Imperfect Reasoning Process](https://mlsanity.com/paper/2501.13536/) (2025)
- [ENTER: Event Based Interpretable Reasoning for VideoQA](https://mlsanity.com/paper/2501.14194/) (2025)
- [DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes](https://mlsanity.com/paper/2505.03581/) (2025)
- [ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling](https://mlsanity.com/paper/2603.22911/) (2026)
- [Scaling Video Understanding via Compact Latent Multi-Agent Collaboration](https://mlsanity.com/paper/2605.00444/) (2026)
- [D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition](https://mlsanity.com/paper/2510.08818/) (2025)
- [Towards Effective Long Video Understanding of Multimodal Large Language Models via One-shot Clip Retrieval](https://mlsanity.com/paper/2512.08410/) (2025)
- [Beyond Isolated Facts: Synthesizing Narrative and Grounded Supervision for VideoQA](https://mlsanity.com/paper/2509.24445/) (2025)
- [VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs](https://mlsanity.com/paper/2512.22226/) (2025)
- [HORNet: Task-Guided Frame Selection for Video Question Answering with Vision-Language Models](https://mlsanity.com/paper/2603.18850/) (2026)
- [CRAFT: Critic-Refined Adaptive Key-Frame Targeting for Multimodal Video Question Answering](https://mlsanity.com/paper/2605.19075/) (2026)
- [Improving Video Question Answering through query-based frame selection](https://mlsanity.com/paper/2601.07459/) (2026)
- [Learning Question-Aware Keyframe Selection with Synthetic Supervision for Video Question Answering](https://mlsanity.com/paper/2603.14953/) (2026)

## Abstract (arXiv)

Video Question Answering (VideoQA) has emerged as a challenging frontier in the field of multimedia processing, requiring intricate interactions between visual and textual modalities. Simply uniformly sampling frames or indiscriminately aggregating frame-level visual features often falls short in capturing the nuanced and relevant contexts of videos to well perform VideoQA. To mitigate these issues, we propose VidF4, a novel VideoQA framework equipped with tailored frame selection strategy for effective and efficient VideoQA. We propose three frame-scoring mechanisms that consider both question relevance and inter-frame similarity to evaluate the importance of each frame for a given question on the video. Furthermore, we design a differentiable adaptive frame sampling mechanism to facilitate end-to-end training for the frame selector and answer generator. The experimental results across three widely adopted benchmarks demonstrate that our model consistently outperforms existing VideoQA methods, establishing a new SOTA across NExT-QA (+0.3%), STAR (+0.9%), and TVQA (+1.0%). Furthermore, through both quantitative and qualitative analyses, we validate the effectiveness of each design choice.