mlsanityData to 7 Oct 2026

LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

arXiv 2505.16933 ↗First posted 22 May 2025cs.LG cs.CL cs.CVcited by 145

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large Language Model (MLLM) that integrates visual instruction tuning with masked diffusion models, representing a departure from the autoregressive paradigms dominant in current multimodal approaches. Built upon LLaDA, a representative large language diffusion model, LLaDA-V incorporates a vision encoder and MLP connector that projects visual features into the language embedding space, enabling effective multimodal alignment. Our empirical investigation reveals several intriguing results: First, LLaDA-V demonstrates promising multimodal performance despite its language model being weaker on purely textual tasks than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal tasks with better data scalability. It also narrows the performance gap to Qwen2-VL, suggesting the effectiveness of its architecture for multimodal tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal understanding compared to existing hybrid autoregressive-diffusion and purely diffusion-based MLLMs. Our findings suggest that large language diffusion models show promise in multimodal contexts and warrant further investigation in future research. Project page and codes: https://ml-gsai.github.io/LLaDA-V-demo/.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 107 of its 114 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

107 of 114 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search13 Mar
  2. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  3. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale6 Dec
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Data3 Oct
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  7. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  9. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
3 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 92

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  14. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  15. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  17. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  18. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  19. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  20. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  21. 2025Large Language Diffusion Modelscited by 872
  22. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  23. 2016A Diagram Is Worth A Dozen Imagescited by 793
  24. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  25. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  26. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  27. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  28. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  29. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  30. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  31. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  32. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  33. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  34. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  35. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  36. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  37. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  38. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 62; 2026: 83.

145 papers.

Sort:

By area:

  1. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  2. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  3. 2025A Survey on Diffusion Language Modelscited by 75
  4. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 64
  5. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  6. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  7. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  8. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  9. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  10. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  11. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  12. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
38 more
  1. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  2. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  3. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  4. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  5. 2026ELF: Embedded Language Flowscited by 27
  6. 2025Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generationcited by 24
  7. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Drivingcited by 24
  8. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 22
  9. 2025Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Waycited by 22
  10. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 22
  11. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  12. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  13. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  14. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  15. 2026Confidence-Based Decoding is Provably Efficient for Diffusion Language Modelscited by 17
  16. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  17. 2025Exploring the Potential of Diffusion Large Language Models in Code Generationcited by 16
  18. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  19. 2025Unveiling the Potential of Diffusion Large Language Model in Controllable Generationcited by 14
  20. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 14
  21. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  22. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 14
  23. 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
  24. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understandingcited by 12
  25. 2025SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understandingcited by 12
  26. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMscited by 12
  27. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 12
  28. 2026Continuous Latent Diffusion Language Modelcited by 12
  29. 2026Improved Large Language Diffusion Modelscited by 12
  30. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 11
  31. 2025CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Creditcited by 11
  32. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  33. 2025WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Drivingcited by 10
  34. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  35. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Modelscited by 10
  36. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
  37. 2025A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Modelscited by 9
  38. 2025Watermarking Diffusion Language Modelscited by 9

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  2. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  5. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
  6. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understanding
  7. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model