mlsanityData to 14 Oct 2026

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

arXiv 2505.16839 ↗First posted 22 May 2025cs.CVcited by 84

On the map: LLM reasoning → Multimodal reasoning

also in 3 more
Abstract (arXiv)

Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models will be released in the camera-ready version.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 73 of its 84 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

73 of 84 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2024GPT-4o System Cardcited by 4,588
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  13. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  14. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  15. 2019Towards VQA Models That Can Readcited by 1,718
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  17. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  18. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  19. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  20. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  21. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  22. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  23. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  24. 2025Large Language Diffusion Modelscited by 912
  25. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  26. 2016A Diagram Is Worth A Dozen Imagescited by 804
  27. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  28. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  29. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  30. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  31. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  32. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  33. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  34. 2021InfographicVQAcited by 437
  35. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  36. 2021OCR-free Document Understanding Transformercited by 407
  37. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  38. 2023Demystifying CLIP Datacited by 270

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 39; 2026: 45.

84 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  3. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  4. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  5. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  6. 2025A Survey on Diffusion Language Modelscited by 78
  7. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  8. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  9. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  10. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  11. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 34
  12. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
38 more
  1. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  2. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 31
  3. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  4. 2025dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thoughtcited by 30
  5. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  6. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  7. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  8. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 19
  9. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  10. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 16
  11. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 15
  12. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  13. 2025SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understandingcited by 13
  14. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 13
  15. 2026dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Modelcited by 13
  16. 2026DOPD: Dual On-policy Distillationcited by 12
  17. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Modelscited by 10
  18. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
  19. 2025A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Modelscited by 9
  20. 2025From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Modelcited by 9
  21. 2026Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLMcited by 9
  22. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 8
  23. 2025Next Semantic Scale Prediction via Hierarchical Diffusion Language Modelscited by 8
  24. 2025MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generationcited by 8
  25. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidancecited by 7
  26. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 7
  27. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 6
  28. 2025Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Modelscited by 6
  29. 2025Watermarking Discrete Diffusion Language Modelscited by 5
  30. 2026Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncationcited by 5
  31. 2025RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruningcited by 4
  32. 2025FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restorationcited by 4
  33. 2026Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inferencecited by 4
  34. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 4
  35. 2026ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusioncited by 4
  36. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Modelscited by 3
  37. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  38. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  2. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
  3. 2025Towards General Continuous Memory for Vision-Language Models
  4. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
  5. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention
  6. 2024Dual Diffusion for Unified Image Generation and Understanding
  7. 2025Emu3.5: Native Multimodal Models are World Learners
  8. 2026Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models