mlsanityData to 7 Oct 2026

DeepSeek-VL: Towards Real-World Vision-Language Understanding

arXiv 2403.05525 ↗First posted 8 Mar 2024cs.AIcited by 718

Not on the map. Why?

Cited by 452 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse, scalable, and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content, aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model's user experience in practical applications. Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024), while maintaining a relatively low computational overhead. This design choice ensures the model's ability to capture critical semantic and detailed information across various visual tasks. We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.

Key facts. Cited by 718 papers in the map (data through 7 Oct 2026). Builds on 62 of its 82 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

62 of 82 references.

References · 62

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
38 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  13. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  17. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  18. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  19. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  20. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  21. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  22. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  23. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  24. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  25. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  26. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  27. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  28. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  29. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  30. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  31. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  32. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  33. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  34. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  35. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  36. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 127
  37. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  38. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 177; 2025: 373; 2026: 168.

718 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  8. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  9. 2024What matters when building vision-language models?cited by 321
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  11. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  12. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
38 more
  1. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  2. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  3. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  4. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  6. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  7. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  8. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  9. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  10. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  11. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  12. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  13. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  14. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  15. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  16. 2026Advancing Open-source World Modelscited by 144
  17. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  18. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  19. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  20. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  21. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  22. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  23. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  24. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  25. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  26. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  27. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  28. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  29. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  30. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  31. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  32. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  33. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  34. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  35. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  36. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  37. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  38. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87

50 most cited of 718 All 718 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Introduction to Vision-Language Modeling
  2. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  4. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  5. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  6. 2025Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models
  7. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions