mlsanityData to 14 Oct 2026

OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

arXiv 2505.04601 ↗First posted 7 May 2025cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

OpenAI's CLIP, released in early 2021, have long been the go-to choice of vision encoder for building multimodal foundation models. Although recent alternatives such as SigLIP have begun to challenge this status quo, to our knowledge none are fully open: their training data remains proprietary and/or their training recipes are not released. This paper fills this gap with OpenVision, a fully-open, cost-effective family of vision encoders that match or surpass the performance of OpenAI's CLIP when integrated into multimodal frameworks like LLaVA. OpenVision builds on existing works -- e.g., CLIPS for training framework and Recap-DataComp-1B for training data -- while revealing multiple key insights in enhancing encoder quality and showcasing practical benefits in advancing multimodal models. By releasing vision encoders spanning from 5.9M to 632.1M parameters, OpenVision offers practitioners a flexible trade-off between capacity and efficiency in building multimodal models: larger models deliver enhanced multimodal performance, while smaller versions enable lightweight, edge-ready multimodal deployments.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

45 of 55 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
33 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  13. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  15. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  16. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  17. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  18. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  19. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  20. 2023Demystifying CLIP Datacited by 270
  21. 2023Data Filtering Networkscited by 250
  22. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  23. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 109
  24. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  25. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  26. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  27. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  28. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  29. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  30. 2025Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And Morecited by 28
  31. 2024CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captionscited by 22
  32. 2023RECLIP: Resource-efficient CLIP by Training with Small Imagescited by 16
  33. 2024Falcon2-11B Technical Reportcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasetscited by 8
  2. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learningcited by 7
  3. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  4. 2026OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generationcited by 5
  5. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  6. 2026MonkeyOCRv2: A Visual-Text Foundation Model for Document AIcited by 4
  7. 2026EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformerscited by 3
  8. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  9. 2025ProCLIP: Progressive Vision-Language Alignment via LLM-based Embeddercited by 1
  10. 2026DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Rolescited by 1
  11. 2026Backdoor Attacks on Multi-modal Contrastive Learningcited by 0
  12. 2026IndicFairFace: Balanced Indian Face Dataset for Auditing and Mitigating Geographical Bias in Vision-Language Modelscited by 0
4 more
  1. 2026InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understandingcited by 0
  2. 2026Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-trainingcited by 0
  3. 2026A Systematic Failure Analysis of Vision Foundation Models for Open Set Iris Presentation Attack Detectioncited by 0
  4. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Modelscited by 0