mlsanityData to 7 Oct 2026

Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

arXiv 2308.12966 ↗First posted 24 Aug 2023cs.CV cs.CLcited by 2,179

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal cleaned corpus. Beyond the conventional image description and question-answering, we implement the grounding and text-reading ability of Qwen-VLs by aligning image-caption-box tuples. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales on a broad range of visual-centric benchmarks (e.g., image captioning, question answering, visual grounding) and different settings (e.g., zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to existing vision-language chatbots. Code, demo and models are available at https://github.com/QwenLM/Qwen-VL.

Key facts. Cited by 2,179 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 86 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

66 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  2. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  5. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  21. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  22. 2016A Diagram Is Worth A Dozen Imagescited by 793
  23. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  24. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  25. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  26. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  27. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  28. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  29. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  30. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  31. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  32. 2018nocaps: novel object captioning at scalecited by 444
  33. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  34. 2021OCR-free Document Understanding Transformercited by 400
  35. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  36. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  37. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  38. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 47; 2024: 598; 2025: 907; 2026: 627.

2,179 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  11. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
38 more
  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  6. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  9. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  10. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  12. 2024What matters when building vision-language models?cited by 321
  13. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  14. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  15. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  16. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  17. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  18. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  19. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  20. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  21. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  22. 2025Motus: A Unified Latent Action World Modelcited by 231
  23. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  24. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  25. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  26. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  27. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  28. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  29. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  30. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  31. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 208
  32. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  33. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  34. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  35. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  36. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  37. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  38. 2024An Introduction to Vision-Language Modelingcited by 175

50 most cited of 2,179 All 2,179 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Interactive Regional Understanding in Vision-Large Language Models
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
  3. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  4. 2025Qwen3-VL Technical Report
  5. 2023Towards Models that Can See and Read
  6. 2026Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
  7. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  8. 2025A Survey on Efficient Vision-Language Models