mlsanityData to 14 Oct 2026

EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

arXiv 2502.06788 ↗First posted 10 Feb 2025cs.CV cs.AIcited by 41

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Existing encoder-free vision-language models (VLMs) are rapidly narrowing the performance gap with their encoder-based counterparts, highlighting the promising potential for unified multimodal systems with structural simplicity and efficient deployment. We systematically clarify the performance gap between VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist visual layers from scratch, deeply excavating the under-examined characteristics of encoder-free VLMs. We develop efficient strategies for encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth investigation, we launch EVEv2.0, a new and improved family of encoder-free VLMs. We show that: (i) Properly decomposing and hierarchically associating vision and language within a unified model reduces interference between modalities. (ii) A well-designed training strategy enables effective optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0 represents a thorough study for developing a decoder-only architecture across modalities, demonstrating superior data efficiency and strong vision-reasoning capability. Code is publicly available at: https://github.com/baaivision/EVE.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 98 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

83 of 98 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding26 Nov
  2. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  3. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  4. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling8 Jul
  5. 2024Unveiling Encoder-Free Vision-Language Models17 Jun
  6. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov

Other references · 73

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2024Qwen2 Technical Reportcited by 2,120
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2019Towards VQA Models That Can Readcited by 1,718
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  14. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  15. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  18. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  20. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  21. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  22. 2016A Diagram Is Worth A Dozen Imagescited by 804
  23. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  24. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  25. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  26. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  27. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  28. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  29. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  30. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  31. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  32. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  33. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  34. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  35. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  36. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  37. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  38. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 20.

41 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  2. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  3. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  4. 2025Vision as LoRAcited by 37
  5. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  6. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  7. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 30
  8. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  9. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  10. 2025OBS-Diff: Accurate Pruning For Diffusion Models in One-Shotcited by 19
  11. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 18
  12. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
29 more
  1. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  2. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?cited by 8
  3. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 7
  4. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraintscited by 6
  5. 2025End-to-End Vision Tokenizer Tuningcited by 5
  6. 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMscited by 5
  7. 2025End-to-end Listen, Look, Speak and Actcited by 5
  8. 2025NeoBabel: A Multilingual Open Tower for Visual Generationcited by 2
  9. 2025MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Expertscited by 2
  10. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 2
  11. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  12. 2026ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMscited by 2
  13. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  14. 2025Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendationcited by 1
  15. 2025VAT: Vision Action Transformer by Unlocking Full Representation of ViTcited by 1
  16. 2026Efficient Encoder-Free Fourier-based 3D Large Multimodal Modelcited by 1
  17. 2026MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQAcited by 1
  18. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  19. 2025Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignmentcited by 0
  20. 2026ELF: A Family of Encoder-Free ECG-Language Modelscited by 0
  21. 2026The Hidden Evolution of Disguised Visual Context inside the VLMcited by 0
  22. 2026SigLIP-HD by Fine-to-Coarse Supervisioncited by 0
  23. 2026NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inferencecited by 0
  24. 2026FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generationcited by 0
  25. 2026Virtual Encoders in Multimodal Transformerscited by 0
  26. 2026ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMscited by 0
  27. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0
  28. 2026Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?cited by 0
  29. 2026Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  3. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
  4. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models
  5. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  6. 2026Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
  7. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  8. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey