mlsanityData to 7 Oct 2026

LLMs Meet Multimodal Generation and Editing: A Survey

arXiv 2405.19334 ↗First posted 29 May 2024cs.AI cs.CL cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning. Previous surveys of multimodal large language models (MLLMs) mainly focus on multimodal understanding. This survey elaborates on multimodal generation and editing across various domains, comprising image, video, 3D, and audio. Specifically, we summarize the notable advancements with milestone works in these fields and categorize these studies into LLM-based and CLIP/T5-based methods. Then, we summarize the various roles of LLMs in multimodal generation and exhaustively investigate the critical technical components behind these methods and the multimodal datasets utilized in these studies. Additionally, we dig into tool-augmented multimodal agents that can leverage existing generative models for human-computer interaction. Lastly, we discuss the advancements in the generative AI safety field, investigate emerging applications, and discuss future prospects. Our work provides a systematic and insightful overview of multimodal generation and processing, which is expected to advance the development of Artificial Intelligence for Generative Content (AIGC) and world models. A curated list of all related papers can be found at https://github.com/YingqingHe/Awesome-LLMs-meet-Multimodal-Generation

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 362 of its 531 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

362 of 531 references.

References · 362

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  12. 2023Segment Anythingcited by 8,158
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2013Auto-Encoding Variational Bayescited by 3,980
  13. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  14. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  15. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  16. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  17. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  18. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  19. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  20. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  21. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  22. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  23. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  24. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  25. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  26. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  27. 2023A Survey of Large Language Modelscited by 2,537
  28. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  29. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  30. 2022Video Diffusion Modelscited by 2,280
  31. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  32. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  33. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  34. 2016Density estimation using Real NVPcited by 2,029
  35. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  36. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  37. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  38. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725

50 most cited of 362 All 362 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 14; 2026: 6.

35 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  3. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  4. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  5. 2024Foundation Models for Music: A Surveycited by 34
  6. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  7. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  8. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 25
  9. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  10. 2024Large Motion Video Autoencoding with Cross-modal Video VAEcited by 19
  11. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  12. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
23 more
  1. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  2. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  3. 2026PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Modelscited by 10
  4. 2024TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimizationcited by 9
  5. 2025Rethinking Layered Graphic Design Generation with a Top-Down Approachcited by 9
  6. 2025Model as a Game: On Numerical and Spatial Consistency for Generative Gamescited by 5
  7. 2025Event-Priori-Based Vision-Language Model for Efficient Visual Understandingcited by 5
  8. 2025UniErase: Towards Balanced and Precise Unlearning in Language Modelscited by 4
  9. 2024ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancementcited by 3
  10. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 2
  11. 2025Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognitioncited by 2
  12. 2025EditInspector: A Benchmark for Evaluation of Text-Guided Image Editscited by 2
  13. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  14. 2025Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Surveycited by 2
  15. 2025Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimizationcited by 1
  16. 2025MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaScited by 1
  17. 2026Modality-Decoupled Online Recursive Editingcited by 1
  18. 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1
  19. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  20. 2025Fake it till You Make it: Reward Modeling as Discriminative Predictioncited by 0
  21. 2026Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editingcited by 0
  22. 2026GraphVid: Interactive Graph-Controllable Video Generationcited by 0
  23. 2026MM-SVGEdit: A Multimodal-Driven SVG Editing for UI Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  4. 2024A Review of Multi-Modal Large Language and Vision Models
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  7. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  8. 2023Multimodal Large Language Models: A Survey