mlsanityData to 7 Oct 2026

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

arXiv 2308.11971 ↗First posted 23 Aug 2023cs.CV cs.CL cs.LGcited by 24

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 3.5x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 64 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

53 of 64 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022VLMAE: Vision-Language Masked Autoencoder19 Aug
  2. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learners15 Jun
  3. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  4. 2022VL-BEiT: Generative Vision-Language Pretraining2 Jun
  5. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  8. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
31 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  13. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  14. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  15. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  16. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  17. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  18. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  19. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  20. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  21. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  22. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  23. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  24. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  25. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  26. 2022Multi-modal Alignment using Representation Codebookcited by 63
  27. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  28. 2021OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generationcited by 20
  29. 2022Training Vision-Language Transformers from Captionscited by 11
  30. 2022MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learningcited by 11
  31. 2023Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  2. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  3. 2025Scaling Laws for Native Multimodal Modelscited by 47
  4. 2024Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicinecited by 28
  5. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  6. 2025HAMoBE: Hierarchical and Adaptive Mixture of Biometric Experts for Video-based Person ReIDcited by 10
  7. 2025DM-Adapter: Domain-Aware Mixture-of-Adapters for Text-Based Person Retrievalcited by 9
  8. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  9. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Expertscited by 5
  10. 2025ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Expertscited by 5
  11. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
  12. 2026On the Holistic Approach for Detecting Human Image Forgerycited by 5
12 more
  1. 2025Enhancing Multi-modal Models with Heterogeneous MoE Adapters for Fine-tuningcited by 3
  2. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  3. 2025MoQE: Improve Quantization Model performance via Mixture of Quantization Expertscited by 3
  4. 2025FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identificationcited by 3
  5. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  6. 2026M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysiscited by 1
  7. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  8. 2026Unleashing MLLMs on the Edge: A Unified Framework for Cross-Modal ReID via Adaptive SVD Distillationcited by 0
  9. 2026ExFusion: Efficient Transformer Training via Multi-Experts Fusioncited by 0
  10. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 0
  11. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0
  12. 2026ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
  2. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
  3. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
  4. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
  5. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
  6. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  7. 2024Unveiling Encoder-Free Vision-Language Models
  8. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling