mlsanityData to 7 Oct 2026

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

arXiv 2411.07975 ↗First posted 12 Nov 2024cs.CV cs.AI cs.CLcited by 162

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We present JanusFlow, a powerful framework that unifies image understanding and generation in a single model. JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. To further improve the performance of our unified model, we adopt two key strategies: (i) decoupling the understanding and generation encoders, and (ii) aligning their representations during unified training. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. This work represents a step toward more efficient and versatile vision-language models.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 86 of its 112 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

86 of 112 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Emu3: Next-Token Prediction is All You Need27 Sep
  2. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding14 May
  3. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment8 Mar
  4. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation27 Feb
  5. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment17 Oct
  6. 2023Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation16 Mar
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Segment Anythingcited by 8,158
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  13. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  15. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  17. 2019Towards VQA Models That Can Readcited by 1,696
  18. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  19. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  20. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  21. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  22. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  23. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  24. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  25. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  26. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  27. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  28. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  29. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  30. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  31. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  32. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  33. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  34. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  35. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  36. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  37. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  38. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 83; 2026: 76.

162 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  4. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 388
  5. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  6. 2025Transfer between Modalities with MetaQueriescited by 231
  7. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  8. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  9. 2025HunyuanImage 3.0 Technical Reportcited by 138
  10. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  11. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  12. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
38 more
  1. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  2. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  3. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  4. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  5. 2025LongCat-Image Technical Reportcited by 92
  6. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  7. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  8. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  9. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  10. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  11. 2025Kling-Omni Technical Reportcited by 60
  12. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  13. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  14. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 51
  15. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  16. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  17. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  18. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 40
  19. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  20. 2025MotionGPT3: Human Motion as a Second Modalitycited by 36
  21. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  22. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  23. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  24. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  25. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  26. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 30
  27. 2026HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformercited by 29
  28. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  29. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  30. 2026Improved Baselines with Representation Autoencoderscited by 24
  31. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
  32. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learningcited by 22
  33. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 21
  34. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  35. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 18
  36. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editingcited by 18
  37. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 17
  38. 2025OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flowscited by 17

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation
  2. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Text
  3. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
  4. 2025UniFusion: Vision-Language Model as Unified Encoder in Image Generation
  5. 2025Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation
  6. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
  7. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
  8. 2026Let ViT Speak: Generative Language-Image Pre-training