mlsanityData to 14 Oct 2026

DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

arXiv 2412.10302 ↗First posted 13 Dec 2024cs.CV cs.AI cs.CLcited by 572

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, through two key major upgrades. For the vision component, we incorporate a dynamic tiling vision encoding strategy designed for processing high-resolution images with different aspect ratios. For the language component, we leverage DeepSeekMoE models with the Multi-head Latent Attention mechanism, which compresses Key-Value cache into latent vectors, to enable efficient inference and high throughput. Trained on an improved vision-language dataset, DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition, document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively. DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models. Codes and pre-trained models are publicly accessible at https://github.com/deepseek-ai/DeepSeek-VL2.

Key facts. Cited by 572 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 112 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

46 of 112 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Pixtral 12B9 Oct
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  6. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception11 Jul
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  9. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 35

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
23 more
  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  2. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  5. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  6. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  7. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  8. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  9. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  10. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  11. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  12. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  13. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  14. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  15. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  16. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  17. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  18. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  19. 2022Syntax-Aware Network for Handwritten Mathematical Expression Recognitioncited by 68
  20. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  21. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  22. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24
  23. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 335; 2026: 233.

572 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  3. 2025Kimi-VL Technical Reportcited by 388
  4. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  5. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  6. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  7. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
  8. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  9. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  10. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 150
  11. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
38 more
  1. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  2. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  4. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  5. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 79
  6. 2025VGR: Visual Grounded Reasoningcited by 73
  7. 2025Detect Anything via Next Point Predictioncited by 73
  8. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 69
  9. 2025Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanationcited by 68
  10. 2025LEGION: Learning to Ground and Explain for Synthetic Image Detectioncited by 65
  11. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  12. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  13. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 55
  14. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  15. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 46
  16. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 41
  17. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  18. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 35
  19. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 34
  20. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  21. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 31
  22. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  23. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 30
  24. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  25. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  26. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  27. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  28. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  29. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 27
  30. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 26
  31. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  32. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 25
  33. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 25
  34. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMscited by 24
  35. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  36. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  37. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  38. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23

50 most cited of 572 All 572 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Scaling Vision-Language Models with Sparse Mixture of Experts
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2026OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing
  4. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  5. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  6. 2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
  7. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  8. 2025Seed1.5-VL Technical Report