mlsanityData to 14 Oct 2026

SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

arXiv 2412.05818 ↗First posted 8 Dec 2024cs.CV cs.AI cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMMs) have demonstrated impressive capabilities in multimodal understanding and generation, pushing forward advancements in text-to-image generation. However, achieving accurate text-image alignment for LMMs, particularly in compositional scenarios, remains challenging. Existing approaches, such as layout planning for multi-step generation and learning from human feedback or AI feedback, depend heavily on prompt engineering, costly human annotations, and continual upgrading, limiting flexibility and scalability. In this work, we introduce a model-agnostic iterative self-improvement framework (SILMM) that can enable LMMs to provide helpful and scalable self-feedback and optimize text-image alignment via Direct Preference Optimization (DPO). DPO can readily applied to LMMs that use discrete visual tokens as intermediate image representations; while it is less suitable for LMMs with continuous visual features, as obtaining generation probabilities is challenging. To adapt SILMM to LMMs with continuous features, we propose a diversity mechanism to obtain diverse representations and a kernel-based continuous DPO for alignment. Extensive experiments on three compositional text-to-image generation benchmarks validate the effectiveness and superiority of SILMM, showing improvements exceeding 30% on T2I-CompBench++ and around 20% on DPG-Bench.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 84 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

62 of 84 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Generative Multimodal Models are In-Context Learners20 Dec
  2. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  3. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  4. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  9. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  13. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  14. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  15. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  16. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  17. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  18. 2024Self-Rewarding Language Modelscited by 681
  19. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  20. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  21. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  22. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  23. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  24. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  25. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  26. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  27. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  28. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  29. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  30. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  31. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  32. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  33. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  34. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  35. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  36. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 101
  37. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  38. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 22
  2. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  3. 2026Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Modelscited by 15
  4. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewardscited by 8
  5. 2025PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Designcited by 6
  6. 2025HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworkscited by 4
  7. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  8. 2025OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generationcited by 1
  9. 2026AUHead: Realistic Emotional Talking Head Generation via Action Units Controlcited by 1
  10. 2026Agentic Flow Steering and Parallel Rollout Search for Spatially Grounded Text-to-Image Generationcited by 1
  11. 2025No Concept Left Behind: Test-Time Optimization for Compositional Text-to-Image Generationcited by 0
  12. 2025EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removalcited by 0
3 more
  1. 2026Training-Free Object-Background Compositional T2I via Dynamic Spatial Guidance and Multi-Path Pruningcited by 0
  2. 2026Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Modelscited by 0
  3. 2026Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization
  2. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2025Aligning Multimodal LLM with Human Preference: A Survey
  5. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
  6. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
  7. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
  8. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer