mlsanityData to 14 Oct 2026

EVLM: An Efficient Vision-Language Model for Visual Understanding

arXiv 2407.14177 ↗First posted 19 Jul 2024cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

In the field of multi-modal language models, the majority of methods are built on an architecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt, directly feeding it into the language models alongside textual tokens. However, when dealing with long sequences of visual signals or inputs such as videos, the self-attention mechanism of language models can lead to significant computational overhead. Additionally, using single-layer ViT features makes it challenging for large language models to perceive visual signals fully. This paper proposes an efficient multi-modal language model to minimize computational costs while enabling the model to perceive visual signals as comprehensively as possible. Our method primarily includes: (1) employing cross-attention to image-text interaction similar to Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of Experts (MoE) mechanism to enhance model effectiveness. Our model achieves competitive scores on public multi-modal benchmarks and performs well in tasks such as image captioning and video captioning.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 58 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

50 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  5. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding3 Mar
  6. 2024CoLLaVO: Crayon Large Language and Vision mOdel17 Feb
  7. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  9. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models21 Dec
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  11. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
2 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
24 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2024Mixtral of Expertscited by 1,823
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  14. 2016A Diagram Is Worth A Dozen Imagescited by 804
  15. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  16. 2024Yi: Open Foundation Models by 01.AIcited by 719
  17. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  18. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  19. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  20. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  22. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  23. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  24. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  2. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  3. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 30
  4. 2024StreamChat: Chatting with Streaming Videocited by 24
  5. 2025Multimodal Language Models See Better When They Look Shallowercited by 23
  6. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 22
  7. 2025VLM as Policy: Common-Law Content Moderation Framework for Short Video Platformcited by 21
  8. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 20
  9. 2024Law of Vision Representation in MLLMscited by 18
  10. 2024Phantom of Latent for Large Language and Vision Modelscited by 17
  11. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 14
  12. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Modelcited by 10
13 more
  1. 2025LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategycited by 10
  2. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  3. 2025RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMscited by 7
  4. 2025TMCIR: Token Merge Benefits Composed Image Retrievalcited by 4
  5. 2025CoMemo: LVLMs Need Image Context with Image Memorycited by 4
  6. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwancited by 3
  7. 2025Vision Remember: Recovering Visual Information in Efficient LVLM with Vision Feature Resamplingcited by 2
  8. 2026Recursive Think-Answer Process for LLMs and VLMscited by 2
  9. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusioncited by 1
  10. 2026From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusioncited by 1
  11. 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
  12. 2026When to Rethink: Learning Multi-Perspective Self-Verification for Vision-Language Modelscited by 0
  13. 2026Sparse All-Layer Connector for Domain Generalised Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  3. 2025Visual Large Language Models for Generalized and Specialized Applications
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  5. 2026Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
  6. 2024An Introduction to Vision-Language Modeling
  7. 2023MLLMs-Augmented Visual-Language Representation Learning
  8. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning