mlsanityData to 14 Oct 2026

EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

arXiv 2211.07636 ↗First posted 14 Nov 2022cs.CV cs.CL cs.LGcited by 691

Not on the map. Why?

Cited by 215 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models. To facilitate future research, we release all the code and models at https://github.com/baaivision/EVA.

Key facts. Cited by 691 papers in the map (data through 14 Oct 2026). Builds on 106 of its 135 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

106 of 135 references.

References · 106

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020Language Models are Few-Shot Learnerscited by 32,419
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  8. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  10. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  11. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  12. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
38 more
  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2016Layer Normalizationcited by 5,669
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  13. 2014Going Deeper with Convolutionscited by 4,241
  14. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  15. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  16. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  17. 2022A ConvNet for the 2020scited by 3,579
  18. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  19. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  20. 2022Emergent Abilities of Large Language Modelscited by 2,512
  21. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  22. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  23. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  24. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  25. 2017The Kinetics Human Action Video Datasetcited by 1,909
  26. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  27. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  28. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  29. 2019Natural Adversarial Examplescited by 1,408
  30. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  31. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  32. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  33. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  34. 2017Random Erasing Data Augmentationcited by 1,239
  35. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  36. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  37. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  38. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 187; 2024: 268; 2025: 169; 2026: 61.

691 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2023A Survey on Multimodal Large Language Modelscited by 909
38 more
  1. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  3. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  4. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  7. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  9. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  10. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  11. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  12. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  13. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  14. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  15. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  16. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  17. 2023Are aligned neural networks adversarially aligned?cited by 315
  18. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 305
  19. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  20. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  21. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  22. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  23. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  24. 2023A Cookbook of Self-Supervised Learningcited by 250
  25. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  26. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  27. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  28. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  29. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  30. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  31. 2023Video Understanding with Large Language Models: A Surveycited by 206
  32. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  33. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  34. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  35. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  36. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  37. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  38. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156

50 most cited of 691 All 691 (JSON)