# EVA-02: A Visual Representation for Neon Genesis

arXiv: https://arxiv.org/abs/2303.11331 · HTML: https://mlsanity.com/paper/2303.11331/

## Key facts

- Cited by 381 papers in the map (data through 14 Oct 2026).
- Builds on 112 of its 147 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 112)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)

## Built on by (most cited first, 30 of 381)

- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024)
- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation](https://mlsanity.com/paper/2503.19755/) (2025)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2504.04348/) (2025)
- [Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders](https://mlsanity.com/paper/2408.15998/) (2024)
- [Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT](https://mlsanity.com/paper/2406.18583/) (2024)
- [DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.07656/) (2025)
- [Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection](https://mlsanity.com/paper/2405.10300/) (2024)
- [4M: Massively Multimodal Masked Modeling](https://mlsanity.com/paper/2312.06647/) (2023)
- [The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World](https://mlsanity.com/paper/2308.01907/) (2023)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters](https://mlsanity.com/paper/2402.04252/) (2024)
- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [Describe Anything: Detailed Localized Image and Video Captioning](https://mlsanity.com/paper/2504.16072/) (2025)
- [Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.04265/) (2023)
- [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://mlsanity.com/paper/2401.10208/) (2024)

## Abstract (arXiv)

We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling. With an updated plain Transformer architecture as well as extensive pre-training from an open & accessible giant CLIP vision encoder, EVA-02 demonstrates superior performance compared to prior state-of-the-art approaches across various representative vision tasks, while utilizing significantly fewer parameters and compute budgets. Notably, using exclusively publicly accessible training data, EVA-02 with only 304M parameters achieves a phenomenal 90.0 fine-tuning top-1 accuracy on ImageNet-1K val set. Additionally, our EVA-02-CLIP can reach up to 80.4 zero-shot top-1 on ImageNet-1K, outperforming the previous largest & best open-sourced CLIP with only ~1/6 parameters and ~1/6 image-text training data. We offer four EVA-02 variants in various model sizes, ranging from 6M to 304M parameters, all with impressive performance. To facilitate open access and open research, we release the complete suite of EVA-02 to the community at https://github.com/baaivision/EVA/tree/master/EVA-02.