# OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation

arXiv: https://arxiv.org/abs/2601.15369 · HTML: https://mlsanity.com/paper/2601.15369/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 52 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation](https://mlsanity.com/paper/2409.04429/) (2024)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)

## Built on by (most cited first, 5 of 5)

- [Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation](https://mlsanity.com/paper/2604.24763/) (2026)
- [WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling](https://mlsanity.com/paper/2605.06407/) (2026)
- [EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers](https://mlsanity.com/paper/2605.16745/) (2026)
- [PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation](https://mlsanity.com/paper/2609.38597/) (2026)
- [FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation](https://mlsanity.com/paper/2609.16591/) (2026)

## Abstract (arXiv)

This paper presents a family of advanced vision encoder, named OpenVision 3, that learns a single, unified visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that synergize and generalize well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder frozen. For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.87 vs. 2.54 on ImageNet). For multimodal understanding, we plug the encoder into the LLaVA-1.5 and LLaVA-NeXT framework: it performs comparably with a standard CLIP vision encoder (e.g., 63.3 vs. 61.2 on SeedBench, and 59.2 vs. 58.1 on GQA). We provide empirical evidence that generation and understanding are mutually beneficial in our architecture, while further underscoring the critical role of the VAE latent space. We hope this work can spur future research on unified modeling.