mlsanityData to 14 Oct 2026

ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

arXiv 2412.06673 ↗First posted 9 Dec 2024cs.CVcited by 64

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 63 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

54 of 63 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  5. 2023Generative Multimodal Models are In-Context Learners20 Dec
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 43

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
31 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  7. 2016A Diagram Is Worth A Dozen Imagescited by 804
  8. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  9. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  10. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  11. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  13. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  14. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  15. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  16. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  17. 2021InfographicVQAcited by 437
  18. 2024OmniGen: Unified Image Generationcited by 394
  19. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  20. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 324
  21. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  22. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  23. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  24. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  25. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  26. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  27. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  28. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesiscited by 44
  29. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 35
  30. 2024PUMA: Empowering Unified MLLM with Multi-granular Visual Generationcited by 20
  31. 2024UNIT: Unifying Image and Text Recognition in One Vision Encodercited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 38; 2026: 26.

64 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  3. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  4. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 109
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  6. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  7. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 56
  8. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  9. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 54
  10. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  11. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 46
  12. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
38 more
  1. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  2. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  3. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 41
  4. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 41
  5. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 36
  6. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 32
  7. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  8. 2025ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RLcited by 30
  9. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 26
  10. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  11. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  12. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  13. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  14. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  15. 2025Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generationcited by 13
  16. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  17. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  18. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 10
  19. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  20. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 8
  21. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  22. 2025Discrete Guidance Matching: Exact Guidance for Discrete Flow Matchingcited by 6
  23. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  24. 2025UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancementcited by 5
  25. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 5
  26. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  27. 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimizationcited by 4
  28. 2026AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Rewardcited by 4
  29. 2026Representation Forcing for Bottleneck-Free Unified Multimodal Modelscited by 4
  30. 2026ChatUMM: Robust Context Tracking for Conversational Interleaved Generationcited by 3
  31. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Taskscited by 3
  32. 2026Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integrationcited by 3
  33. 2026Dreaming in Flow: Generative Grounding Feedback for Self-Evolving Unified Multimodal Modelscited by 3
  34. 2025Resolving Task Objective Conflicts in Unified Model via Task-Aware Mixture-of-Expertscited by 2
  35. 2026EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generationcited by 2
  36. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  37. 2026STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generationcited by 2
  38. 2026ICG: Improving Cover Image Generation via MLLM-based Prompting and Personalized Preference Alignmentcited by 2

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation
  2. 2023LLMGA: Multimodal Large Language Model based Generation Assistant
  3. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
  4. 2023Planting a SEED of Vision in Large Language Model
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  6. 2024Semantic Alignment for Multimodal Large Language Models
  7. 2024The Revolution of Multimodal Large Language Models: A Survey
  8. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs