mlsanityData to 7 Oct 2026

Perception Test: A Diagnostic Benchmark for Multimodal Video Models

arXiv 2305.13786 ↗First posted 23 May 2023cs.CV cs.AI cs.LGcited by 310

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baseline code, and challenge server are available at https://github.com/deepmind/perception_test

Key facts. Cited by 310 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 60 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

39 of 60 references.

References · 39

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  7. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
27 more
  1. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  2. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  3. 2020Rescaling Egocentric Visioncited by 529
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  5. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  6. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  7. 2022Kubric: A scalable dataset generatorcited by 373
  8. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  9. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  10. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  11. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  12. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  13. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  14. 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 185
  15. 2022Multiview Transformers for Video Recognitioncited by 148
  16. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  17. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  18. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  19. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  20. 2020LaSOT: A High-quality Large-scale Single Object Tracking Benchmarkcited by 130
  21. 2020TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Taskscited by 77
  22. 2018IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoningcited by 72
  23. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  24. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  25. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  26. 2015Visual object tracking performance measures revisitedcited by 12
  27. 2014Towards a Visual Turing Challengecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 80; 2025: 134; 2026: 88.

310 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  7. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  8. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  9. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  10. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  11. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  12. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
38 more
  1. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  2. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  3. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  4. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  5. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  6. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  7. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  8. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  9. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  10. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  11. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  12. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  13. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  14. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  15. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  16. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  17. 2025Scaling RL to Long Videoscited by 83
  18. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  19. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  20. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  21. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  22. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  23. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  24. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  25. 2024Needle In A Multimodal Haystackcited by 56
  26. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  27. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 55
  28. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  29. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  30. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  31. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  32. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  33. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 43
  34. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
  35. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  36. 2024Baichuan-Omni Technical Reportcited by 42
  37. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answeringcited by 40
  38. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37

50 most cited of 310 All 310 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs
  2. 2026MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding
  3. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning
  4. 2026How Far Are Video Models from True Multimodal Reasoning?
  5. 2026PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning
  6. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  7. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
  8. 2025All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark