mlsanityData to 14 Oct 2026

MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

arXiv 2412.14164 ↗First posted 18 Dec 2024cs.CVcited by 206

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we propose Visual-Predictive Instruction Tuning (VPiT) - a simple and effective extension to visual instruction tuning that enables a pretrained LLM to quickly morph into an unified autoregressive model capable of generating both text and visual tokens. VPiT teaches an LLM to predict discrete text tokens and continuous visual tokens from any input sequence of image and text data curated in an instruction-following format. Our empirical investigation reveals several intriguing properties of VPiT: (1) visual generation ability emerges as a natural byproduct of improved visual understanding, and can be unlocked efficiently with a small amount of generation data; (2) while we find understanding and generation to be mutually beneficial, understanding data contributes to both capabilities more effectively than generation data. Building upon these findings, we train our MetaMorph model and achieve competitive performance on both visual understanding and generation. In visual generation, MetaMorph can leverage the world knowledge and reasoning abilities gained from LLM pretraining, and overcome common failure modes exhibited by other generation models. Our results suggest that LLMs may have strong "prior" vision capabilities that can be efficiently adapted to both visual understanding and generation with a relatively simple instruction tuning process.

Key facts. Cited by 206 papers in the map (data through 14 Oct 2026). Builds on 72 of its 85 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

72 of 85 references.

References · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2016Layer Normalizationcited by 5,669
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  13. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  15. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  18. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  19. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  20. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  21. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  22. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  23. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  24. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  25. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  26. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  27. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  28. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  29. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  30. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  31. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  32. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  33. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  34. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  35. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  36. 2024The Platonic Representation Hypothesiscited by 382
  37. 2024What matters when building vision-language models?cited by 326
  38. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 123; 2026: 83.

206 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  3. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  4. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  5. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  6. 2025Transfer between Modalities with MetaQueriescited by 236
  7. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  8. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  9. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  10. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  11. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  12. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
38 more
  1. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 124
  2. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  3. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 109
  4. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  5. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  6. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  7. 2025Scaling Language-Free Visual Representation Learningcited by 66
  8. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 66
  9. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  10. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 58
  11. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 57
  12. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  13. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
  14. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  15. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
  16. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 53
  17. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  18. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  19. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  20. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  21. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 46
  22. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  23. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  24. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  25. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 41
  26. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 41
  27. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 40
  28. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  29. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  30. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 33
  31. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  32. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 32
  33. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 32
  34. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  35. 2025ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RLcited by 30
  36. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 26
  37. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 24
  38. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Modelscited by 23

50 most cited of 206 All 206 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generative Visual Instruction Tuning
  2. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  3. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning
  4. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  5. 2025Learning to Instruct for Visual Instruction Tuning
  6. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
  7. 2026Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Models
  8. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data