mlsanityData to 7 Oct 2026

OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation

arXiv 2509.03498 ↗First posted 3 Sep 2025cs.CVcited by 50

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

We introduce OneCAT, a unified multimodal model that seamlessly integrates understanding, generation, and editing within a novel, pure decoder-only transformer architecture. Our framework uniquely eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference, leading to significant efficiency gains, especially for high-resolution inputs. This is achieved through a modality-specific Mixture-of-Experts (MoE) structure trained with a single autoregressive (AR) objective, which also natively supports dynamic resolutions. Furthermore, we pioneer a multi-scale visual autoregressive mechanism within the Large Language Model (LLM) that drastically reduces decoding steps compared to diffusion-based methods while maintaining state-of-the-art performance. Our findings demonstrate the powerful potential of pure autoregressive modeling as a sufficient and elegant foundation for unified multimodal intelligence. As a result, OneCAT sets a new performance standard, outperforming existing open-source unified multimodal models across benchmarks for multimodal generation, editing, and understanding.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 97 of its 111 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

97 of 111 references.

References · 97

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  15. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  18. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  20. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  21. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  22. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  23. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  24. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  25. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  26. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  27. 2023A Survey on Multimodal Large Language Modelscited by 898
  28. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  29. 2016A Diagram Is Worth A Dozen Imagescited by 793
  30. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  31. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  32. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  33. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  34. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  35. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  36. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  37. 2021InfographicVQAcited by 430
  38. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 37.

50 papers.

Sort:

By area:

  1. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  2. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  3. 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
  4. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  5. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  6. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  7. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
  8. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 23
  9. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  10. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
  11. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 15
  12. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 13
38 more
  1. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  2. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11
  3. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 11
  4. 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 9
  5. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 9
  6. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  7. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 7
  8. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  9. 2026Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modelingcited by 6
  10. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  11. 2026Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformerscited by 4
  12. 2026Uni-Edit: Intelligent Editing Is A General Task For Unified Model Tuningcited by 4
  13. 2026MetaPoint: Unlocking Precise Spatial Control in Agentic Visual Generationcited by 4
  14. 2026UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformercited by 4
  15. 2026PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Taskscited by 3
  16. 2026Steering Visual Generation in Unified Multimodal Models with Understanding Supervisioncited by 3
  17. 2026Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasonerscited by 3
  18. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 3
  19. 2025UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmarkcited by 2
  20. 2026A Unified Language Model for Large Scale Search, Recommendation, and Reasoningcited by 2
  21. 2026STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generationcited by 2
  22. 2026HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizerscited by 2
  23. 2025AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Modelcited by 1
  24. 2025WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokenscited by 1
  25. 2026Hierarchical Concept-to-Appearance Guidance for Multi-Subject Image Generationcited by 1
  26. 2026Mobile-O: Unified Multimodal Understanding and Generation on Mobile Devicecited by 1
  27. 2026HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modelingcited by 1
  28. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  29. 2026Protein Autoregressive Modeling via Multiscale Structure Generationcited by 0
  30. 2026StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Modelscited by 0
  31. 2026A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Predictioncited by 0
  32. 2026How Long Can Unified Multimodal Models Generate Images Reliably? Taming Long-Horizon Interleaved Image Generation via Context Curationcited by 0
  33. 2026Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Modelcited by 0
  34. 2026UniVR: Thinking in Visual Space for Unified Visual Reasoningcited by 0
  35. 2026STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMscited by 0
  36. 2026SciGen-Verifier: A Multimodal Reasoner for Explainable Verification in Scientific Image Generationcited by 0
  37. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0
  38. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation
  2. 2026UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
  3. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  4. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  5. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation
  6. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
  7. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
  8. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer