mlsanityData to 14 Oct 2026

PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

arXiv 2504.13180 ↗First posted 17 Apr 2025cs.CV cs.AI cs.LGcited by 92

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Vision-language models are integral to computer vision research, yet many high-performing models remain closed-source, obscuring their data, design and training recipe. The research community has responded by using distillation from black-box models to label training data, achieving strong benchmark results, at the cost of measurable scientific progress. However, without knowing the details of the teacher model and its data sources, scientific progress remains difficult to measure. In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding. We analyze standard training pipelines without distillation from proprietary models and explore large-scale synthetic data to identify critical data gaps, particularly in detailed video understanding. To bridge these gaps, we release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded video captions. Additionally, we introduce PLM-VideoBench, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about "what", "where", "when", and "how" of a video. We make our work fully reproducible by providing data, training recipes, code & models. https://github.com/facebookresearch/perception_models

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 17 Apr 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 34; 2026: 58.

92 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
  3. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  4. 2025Visual Spatial Tuningcited by 89
  5. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 84
  6. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  7. 2025FineVision: Open Data Is All You Needcited by 53
  8. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
  9. 2026Revisiting the Platonic Representation Hypothesis: An Aristotelian Viewcited by 44
  10. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  11. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  12. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 30
38 more
  1. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  2. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 25
  3. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  4. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17
  5. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 16
  6. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  7. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  8. 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
  9. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  10. 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizerscited by 13
  11. 2026Action100M: A Large-scale Video Action Datasetcited by 12
  12. 2026SAMTok: Representing Any Mask with Two Wordscited by 11
  13. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  14. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 10
  15. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lenscited by 9
  16. 2025ExAct: A Video-Language Benchmark for Expert Action Analysiscited by 8
  17. 2025Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-incited by 8
  18. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMscited by 7
  19. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 7
  20. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  21. 2025NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filteringcited by 6
  22. 2025FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videoscited by 5
  23. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  24. 2025Counterfactual World Models via Digital Twin-conditioned Video Diffusioncited by 4
  25. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 4
  26. 2026Motion-o: Trajectory-Grounded Video Reasoningcited by 4
  27. 2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Taskscited by 4
  28. 2026Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programscited by 4
  29. 2026Don't Pause! Every prediction matters in a streaming videocited by 4
  30. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  31. 2025VUGEN: Visual Understanding priors for GENerationcited by 3
  32. 2026NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understandingcited by 3
  33. 2026EventFlash: Towards Efficient MLLMs for Event-Based Visioncited by 3
  34. 2026Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervisioncited by 3
  35. 2026Can Vision-Language Models Solve the Shell Game?cited by 3
  36. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  37. 2025Mixture of States: Routing Token-Level Dynamics for Multimodal Generationcited by 2
  38. 2026ZAYA1-VL-8B Technical Reportcited by 2

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  2. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  3. 2025GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?
  4. 2026Watch Before You Answer: Learning from Visually Grounded Post-Training
  5. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  6. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs