mlsanityData to 14 Oct 2026

Auto-Encoding Morph-Tokens for Multimodal LLM

arXiv 2405.01926 ↗First posted 3 May 2024cs.CVcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge. This is due to a conflicting objective: for comprehension, an MLLM needs to abstract the visuals; for generation, it needs to preserve the visuals as much as possible. Thus, the objective is a dilemma for visual-tokens. To resolve the conflict, we propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts; for generation, they take on a different, non-conflicting role as complete visual-tokens for image reconstruction, where the missing visual cues are recovered by the MLLM. Extensive experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously. Our project is available at https://github.com/DCDmllm/MorphTokens.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 59 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

49 of 59 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  2. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Models9 Feb
  3. 2023Generative Multimodal Models are In-Context Learners20 Dec
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  5. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  6. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions8 Aug
  7. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023Generating Images with Multimodal Language Models26 May
  10. 2023Equivariant Similarity for Vision-Language Foundation Models25 Mar

Other references · 39

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
27 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  2. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  3. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  4. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  5. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  6. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  7. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  9. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  10. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  13. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  14. 2022Visual Spatial Reasoningcited by 293
  15. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  16. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  17. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  18. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  19. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  20. 2020Visual Commonsense R-CNNcited by 106
  21. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  22. 2019Expressing Visual Relationships via Languagecited by 49
  23. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  24. 2022UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidancecited by 30
  25. 2021Learning by Planning: Language-Guided Global Image Editingcited by 25
  26. 2024RECOST: External Knowledge Guided Data-efficient Instruction Tuningcited by 18
  27. 2023Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalizationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 16; 2026: 2.

30 papers.

Sort:

By area:

  1. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  2. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 63
  3. 2024GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphscited by 58
  4. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 56
  5. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  6. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 46
  7. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  8. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 33
  9. 2024HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editingcited by 29
  10. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 28
  11. 2026OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoningcited by 27
  12. 2024Towards Unified Multimodal Editing with Enhanced Knowledge Collaborationcited by 18
18 more
  1. 2025Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learningcited by 18
  2. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  3. 2025FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RLcited by 13
  4. 2024GPT-4o: Visual perception performance of multimodal large language models in piglet activity understandingcited by 12
  5. 2025EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editingcited by 12
  6. 2025WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editingcited by 11
  7. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  8. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  9. 2025Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learningcited by 9
  10. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  11. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  12. 2025Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLMcited by 7
  13. 2025The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Modelscited by 3
  14. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  15. 2025FOUNDv2: Learning Unified User Quantized Tokenizers for User Representationcited by 1
  16. 2025Towards Meta-Cognitive Knowledge Editing for Multimodal LLMscited by 1
  17. 2025OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructionscited by 1
  18. 2026Detached Skip-Links and R-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  2. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs
  3. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  4. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM
  5. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  6. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  7. 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task
  8. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation