# PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

arXiv: https://arxiv.org/abs/2504.13180 · HTML: https://mlsanity.com/paper/2504.13180/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 17 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Built on by (most cited first, 30 of 92)

- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [Visual Spatial Tuning](https://mlsanity.com/paper/2511.05491/) (2025)
- [What matters for Representation Alignment: Global Information or Spatial Structure?](https://mlsanity.com/paper/2512.10794/) (2025)
- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [FineVision: Open Data Is All You Need](https://mlsanity.com/paper/2510.17269/) (2025)
- [Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence](https://mlsanity.com/paper/2510.20579/) (2025)
- [Revisiting the Platonic Representation Hypothesis: An Aristotelian View](https://mlsanity.com/paper/2602.14486/) (2026)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models](https://mlsanity.com/paper/2506.09943/) (2025)
- [Seeing the Arrow of Time in Large Multimodal Models](https://mlsanity.com/paper/2506.03340/) (2025)
- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos](https://mlsanity.com/paper/2506.05349/) (2025)
- [Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs](https://mlsanity.com/paper/2510.18876/) (2025)
- [OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows](https://mlsanity.com/paper/2510.03506/) (2025)
- [When Thinking Drifts: Evidential Grounding for Robust Video Reasoning](https://mlsanity.com/paper/2510.06077/) (2025)
- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [MolmoPoint: Better Pointing for VLMs with Grounding Tokens](https://mlsanity.com/paper/2603.28069/) (2026)
- [HoneyBee: Data Recipes for Vision-Language Reasoners](https://mlsanity.com/paper/2510.12225/) (2025)
- [Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning](https://mlsanity.com/paper/2512.19687/) (2025)
- [RepWAM: World Action Modeling with Representation Visual-Action Tokenizers](https://mlsanity.com/paper/2606.13674/) (2026)
- [Action100M: A Large-scale Video Action Dataset](https://mlsanity.com/paper/2601.10592/) (2026)
- [SAMTok: Representing Any Mask with Two Words](https://mlsanity.com/paper/2601.16093/) (2026)
- [TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs](https://mlsanity.com/paper/2602.00288/) (2026)
- [VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding](https://mlsanity.com/paper/2607.14935/) (2026)
- [From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens](https://mlsanity.com/paper/2510.02292/) (2025)
- [ExAct: A Video-Language Benchmark for Expert Action Analysis](https://mlsanity.com/paper/2506.06277/) (2025)
- [Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-in](https://mlsanity.com/paper/2512.14273/) (2025)
- [SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs](https://mlsanity.com/paper/2602.06566/) (2026)

## Abstract (arXiv)

Vision-language models are integral to computer vision research, yet many high-performing models remain closed-source, obscuring their data, design and training recipe. The research community has responded by using distillation from black-box models to label training data, achieving strong benchmark results, at the cost of measurable scientific progress. However, without knowing the details of the teacher model and its data sources, scientific progress remains difficult to measure. In this paper, we study building a Perception Language Model (PLM) in a fully open and reproducible framework for transparent research in image and video understanding. We analyze standard training pipelines without distillation from proprietary models and explore large-scale synthetic data to identify critical data gaps, particularly in detailed video understanding. To bridge these gaps, we release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded video captions. Additionally, we introduce PLM-VideoBench, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about "what", "where", "when", and "how" of a video. We make our work fully reproducible by providing data, training recipes, code & models. https://github.com/facebookresearch/perception_models