mlsanityData to 7 Oct 2026

Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing

arXiv 2603.12254 ↗First posted 12 Mar 2026cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 89 of its 110 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

89 of 110 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025FOCUS: Efficient Keyframe Selection for Long Video Understanding31 Oct
  2. 2025Scaling RL to Long Videos10 Jul
  3. 2025Improving LLM Video Understanding with 16 Frames Per Second18 Mar
  4. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMs6 Mar
  5. 2025Adaptive Keyframe Sampling for Long Video Understanding28 Feb
  6. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling31 Dec
  7. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Models13 Dec
  8. 2024VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation1 Dec
  9. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  10. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  11. 2024Long Context Transfer from Language to Vision24 Jun
  12. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
2 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov

Other references · 75

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2018SlowFast Networks for Video Recognitioncited by 1,862
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  10. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2022Token Merging: Your ViT But Fastercited by 995
  13. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  14. 2021Multiscale Vision Transformerscited by 896
  15. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  16. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  17. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  18. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  19. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  20. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  21. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  22. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  23. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  24. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  25. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  26. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  27. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  28. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  29. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  30. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  31. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  32. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  33. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  34. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  35. 2021NeRV: Neural Representations for Videoscited by 255
  36. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  37. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  38. 2019Video Representation Learning by Dense Predictive Codingcited by 175

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMscited by 2
  2. 2026LookWhen? Fast Video Recognition by Learning When, Where, and What to Computecited by 1
  3. 2026EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMscited by 1
  4. 2026Policy-based Foveated Imaging and Perceptioncited by 0
  5. 2026Stateful Visual Encoders for Vision-Language Modelscited by 0
  6. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  7. 2026KeyRec: Bounded Visual Memory for Streaming and Long-Video Understandingcited by 0
  8. 2026MotionInsight: Diagnosing Object Motion Deficiencies in Generated Videoscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashGaze: Training-Free Multi-Scale Patch Pruning For Efficient Video Understanding
  2. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  3. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  4. 2026Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
  5. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
  6. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding
  7. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
  8. 2026VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning