mlsanityData to 7 Oct 2026

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

arXiv 2410.13848 ↗First posted 17 Oct 2024cs.CV cs.AI cs.CLcited by 447

Not on the map. Why?

Cited by 91 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often relies on a single visual encoder for both tasks, such as Chameleon. However, due to the differing levels of information granularity required by multimodal understanding and generation, this approach can lead to suboptimal performance, particularly in multimodal understanding. To address this issue, we decouple visual encoding into separate pathways, while still leveraging a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the visual encoder's roles in understanding and generation, but also enhances the framework's flexibility. For instance, both the multimodal understanding and generation components can independently select their most suitable encoding methods. Experiments show that Janus surpasses previous unified model and matches or exceeds the performance of task-specific models. The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.

Key facts. Cited by 447 papers in the map (data through 7 Oct 2026). Builds on 75 of its 98 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

75 of 98 references.

References · 75

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2023Segment Anythingcited by 8,158
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  13. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  14. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  15. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  17. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  18. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  19. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  20. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  21. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  22. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  23. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  24. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  25. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  26. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  27. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  28. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  29. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  30. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  31. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  32. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  33. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  34. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  35. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  36. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  37. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  38. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 238; 2026: 186.

447 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  5. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  7. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  8. 2025Transfer between Modalities with MetaQueriescited by 231
  9. 2025Motus: A Unified Latent Action World Modelcited by 231
  10. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  11. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  12. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
38 more
  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  3. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  4. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  5. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  6. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  7. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  8. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  9. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  10. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  11. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  12. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  13. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  14. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  15. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  16. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  17. 2025LongCat-Image Technical Reportcited by 92
  18. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  19. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  20. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  21. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  22. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  23. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  24. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  25. 2025A Survey on Diffusion Language Modelscited by 75
  26. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  27. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  28. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  29. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  30. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  31. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  32. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  33. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  34. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  35. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  36. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  37. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  38. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50

50 most cited of 447 All 447 (JSON)