mlsanityData to 14 Oct 2026

Harmonizing Visual Representations for Unified Multimodal Understanding and Generation

arXiv 2503.21979 ↗First posted 27 Mar 2025cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

Unifying visual understanding and generation within a single multimodal framework remains a significant challenge, as the two inherently heterogeneous tasks require representations at different levels of granularity. Current approaches that utilize vector quantization (VQ) or variational autoencoders (VAE) for unified visual representation prioritize intrinsic imagery features over semantics, compromising understanding performance. In this work, we take inspiration from masked image modelling (MIM) that learns rich semantics via a mask-and-reconstruct pre-training and its successful extension to masked autoregressive (MAR) image generation. A preliminary study on the MAR encoder's representation reveals exceptional linear probing accuracy and precise feature response to visual concepts, which indicates MAR's potential for visual understanding tasks beyond its original generation role. Based on these insights, we present Harmon, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder. Through a three-stage training procedure that progressively optimizes understanding and generation capabilities, Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks while matching the performance of methods with dedicated semantic encoders (e.g., Janus) on image understanding benchmarks. Our code and models will be available at https://github.com/wusize/Harmon.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Builds on 81 of its 100 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

81 of 100 references.

References · 81

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2024Qwen2.5 Technical Reportcited by 4,942
  12. 2017Neural Discrete Representation Learningcited by 4,712
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2022A ConvNet for the 2020scited by 3,579
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  13. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  14. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  15. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  17. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  18. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  19. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  21. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  22. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  23. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  24. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  25. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  26. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  27. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  28. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  29. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  30. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  31. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  32. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  33. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  34. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  35. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  36. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  37. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  38. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 33; 2026: 41.

74 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  2. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  3. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  4. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  5. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  6. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 46
  7. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  8. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 41
  9. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  10. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  11. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  12. 2026Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generationcited by 20
38 more
  1. 2025Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generationcited by 19
  2. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  3. 2025FARMER: Flow AutoRegressive Transformer over Pixelscited by 18
  4. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  5. 2025Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generationcited by 17
  6. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 17
  7. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  8. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  9. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  10. 2026UniReason 1.0: A Unified Reasoning Framework for World Knowledge Aligned Image Generation and Editingcited by 12
  11. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  12. 2025Locality-aware Parallel Decoding for Efficient Autoregressive Image Generationcited by 10
  13. 2025Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMscited by 10
  14. 2025OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilitiescited by 9
  15. 2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
  16. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  17. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 8
  18. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  19. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 7
  20. 2025Plug-and-Play Context Feature Reuse for Efficient Masked Generationcited by 5
  21. 2025DiSA: Diffusion Step Annealing in Autoregressive Image Generationcited by 5
  22. 2026Scaling GUI Agents with Visual State Transitionscited by 4
  23. 2025Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Maskingcited by 3
  24. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  25. 2026Multi-Scale Local Speculative Decoding for Image Generationcited by 3
  26. 2026VIGIL: Part-Grounded Structured Reasoning for Generalizable Deepfake Detectioncited by 3
  27. 2026UniGenDet: A Unified Generative-Discriminative Framework for Co-Evolutionary Image Generation and Generated Image Detectioncited by 3
  28. 2026Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasonerscited by 3
  29. 2026Transferability Between Understanding and Generation in Unified Multimodal Modelscited by 3
  30. 2025Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Samplingcited by 2
  31. 2026IMTBench: A Multi-Scenario Cross-Modal Collaborative Evaluation Benchmark for In-Image Machine Translationcited by 2
  32. 2026HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understandingcited by 2
  33. 2026Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-trainingcited by 2
  34. 2026MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generationcited by 2
  35. 2026Camera Control for Text-to-Image Generation via Learning Viewpoint Tokenscited by 2
  36. 2026Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generationcited by 2
  37. 2026HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizerscited by 2
  38. 2026Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to Systemcited by 2

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification
  2. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
  3. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  4. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
  5. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better
  6. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
  7. 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
  8. 2025VUGEN: Visual Understanding priors for GENeration