mlsanityData to 14 Oct 2026

PUMA: Empowering Unified MLLM with Multi-granular Visual Generation

arXiv 2410.13861 ↗First posted 17 Oct 2024cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Recent advancements in multimodal foundation models have yielded significant progress in vision-language understanding. Initial attempts have also explored the potential of multimodal large language models (MLLMs) for visual content generation. However, existing works have insufficiently addressed the varying granularity demands of different image generation tasks within a unified MLLM paradigm - from the diversity required in text-to-image generation to the precise controllability needed in image manipulation. In this work, we propose PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA unifies multi-granular visual features as both inputs and outputs of MLLMs, elegantly addressing the different granularity requirements of various image generation tasks within a unified MLLM framework. Following multimodal pretraining and task-specific instruction tuning, PUMA demonstrates proficiency in a wide range of multimodal tasks. This work represents a significant step towards a truly unified MLLM capable of adapting to the granularity demands of various visual tasks. The code and model will be released in https://github.com/rongyaofang/PUMA.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 56 of its 71 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

56 of 71 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  13. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  16. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  18. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  19. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  20. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  21. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  22. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  23. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  24. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  25. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  26. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  27. 2024OmniGen: Unified Image Generationcited by 394
  28. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  29. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  30. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 324
  31. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  32. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  33. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  34. 2023Generating Images with Multimodal Language Modelscited by 260
  35. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  36. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  37. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
  38. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 1.

20 papers.

Sort:

By area:

  1. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  2. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 104
  3. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 99
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  5. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  6. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 54
  7. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 52
  8. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 38
  9. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  10. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  11. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  12. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
8 more
  1. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  2. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  3. 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 10
  4. 2025V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generationcited by 6
  5. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 5
  6. 2025DreamOmni3: Scribble-based Editing and Generationcited by 2
  7. 2026DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Modelscited by 1
  8. 2025UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  4. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  5. 2024An Introduction to Vision-Language Modeling
  6. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM
  7. 2026PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs
  8. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?