mlsanityData to 14 Oct 2026

Perception Encoder: The best visual embeddings are not at the output of the network

arXiv 2504.13181 ↗First posted 17 Apr 2025cs.CVcited by 326

Not on the map. Why?

Cited by 65 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Perception Encoder (PE), a state-of-the-art vision encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, our PE family of models achieves best-in-class results on a wide variety of tasks, including (1) zero-shot image and video classification and retrieval, simultaneously obtaining 86.6 average zero-shot ImageNet robustness and 76.9 zero-shot Kinetics-400 video classification; (2) document, image, and video Q&A, enabling 94.6 DocVQA, 80.9 InfographicVQA, and 82.7 PerceptionTest with an 8B LLM; and (3) spatial tasks such as detection, tracking, and depth estimation, setting a new COCO state-of-the-art of 66.0 box mAP. To foster further research, we release our models, code, and novel dataset of synthetically and human-annotated videos: https://github.com/facebookresearch/perception_models

Key facts. Cited by 326 papers in the map (data through 14 Oct 2026). Builds on 135 of its 164 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

135 of 164 references.

References · 135

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  8. 2024The Llama 3 Herd of Modelscited by 14,715
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  12. 2023Segment Anythingcited by 8,234
38 more
  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2017Mask R-CNNcited by 7,277
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  13. 2022A ConvNet for the 2020scited by 3,579
  14. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  15. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  16. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  17. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  18. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  19. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  20. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  21. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  22. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  23. 2021Vision Transformers for Dense Predictioncited by 1,803
  24. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  25. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  26. 2019Towards VQA Models That Can Readcited by 1,718
  27. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  28. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  29. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  30. 2019Natural Adversarial Examplescited by 1,408
  31. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  32. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  33. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  34. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  35. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  36. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  37. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  38. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188

50 most cited of 135 All 135 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 104; 2026: 222.

326 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  3. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  4. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 84
  5. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  6. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  7. 2026Revisiting the Platonic Representation Hypothesis: An Aristotelian Viewcited by 44
  8. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  9. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  10. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  11. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  12. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
38 more
  1. 2026STEP3-VL-10B Technical Reportcited by 32
  2. 2026Improved Baselines with Representation Autoencoderscited by 31
  3. 2025AToken: A Unified Tokenizer for Visioncited by 30
  4. 2026VGGT-SLAM 2.0: Real-time Dense Feed-forward Scene Reconstructioncited by 28
  5. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  6. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  7. 2025E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-trainingcited by 22
  8. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  9. 2026Interpreting Physics in Video World Modelscited by 22
  10. 2025Deepfake Detection that Generalizes Across Benchmarkscited by 21
  11. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 20
  12. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 20
  13. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 19
  14. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  15. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  16. 2025VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Modelscited by 17
  17. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 16
  18. 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 16
  19. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  20. 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
  21. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 13
  22. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  23. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  24. 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizerscited by 13
  25. 2025Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detectioncited by 12
  26. 2025Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teacherscited by 12
  27. 2026Action100M: A Large-scale Video Action Datasetcited by 12
  28. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 12
  29. 2026DM0: An Embodied-Native Vision-Language-Action Model towards Physical AIcited by 12
  30. 2026INSID3: Training-Free In-Context Segmentation with DINOv3cited by 12
  31. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understandingcited by 11
  32. 2026Representation Distribution Matching for One-Step Visual Generationcited by 11
  33. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  34. 2025Brought a Gun to a Knife Fight: Modern VFM Baselines Outgun Specialized Detectors on In-the-Wild AI Image Detectioncited by 10
  35. 2025Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encodingcited by 10
  36. 2026Efficient Universal Perception Encodercited by 10
  37. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 10
  38. 2025No time to train! Training-Free Reference-Based Instance Segmentationcited by 9

50 most cited of 326 All 326 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  2. 2024Unveiling Encoder-Free Vision-Language Models
  3. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
  4. 2025Interpreting the linear structure of vision-language model embedding spaces
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  6. 2023Joint Adaptive Representations for Image-Language Learning
  7. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
  8. 2026Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs