mlsanityData to 14 Oct 2026

Unified Autoregressive Visual Generation and Understanding with Continuous Tokens

arXiv 2503.13436 ↗First posted 17 Mar 2025cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We present UniFluid, a unified autoregressive framework for joint visual generation and understanding leveraging continuous visual tokens. Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image. We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other. By selecting an appropriate loss balance weight, the unified model achieves results comparable to or exceeding those of single-task baselines on both tasks. Furthermore, we demonstrate that employing stronger pre-trained LLMs and random-order generation during training is important to achieve high-fidelity image generation within this unified framework. Built upon the Gemma model series, UniFluid exhibits competitive performance across both image generation and understanding, demonstrating strong transferability to various downstream tasks, including image editing for generation, as well as visual captioning and question answering for understanding.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 55 of its 61 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

55 of 61 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  11. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  13. 2016Pixel Recurrent Neural Networkscited by 816
  14. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  15. 2016A Diagram Is Worth A Dozen Imagescited by 804
  16. 2018Image Transformercited by 772
  17. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  18. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  19. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  20. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  21. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  22. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  23. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  24. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  25. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  26. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  27. 2021InfographicVQAcited by 437
  28. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  29. 2019Scene Text Visual Question Answeringcited by 331
  30. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  31. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  32. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  33. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 203
  34. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
  35. 2018TallyQA: Answering Complex Counting Questionscited by 153
  36. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  37. 2024Randomized Autoregressive Visual Generationcited by 127
  38. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 24.

51 papers.

Sort:

By area:

  1. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  2. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  4. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 58
  5. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 57
  6. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 53
  7. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  8. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 40
  9. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  10. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  11. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  12. 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
38 more
  1. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  2. 2026Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generationcited by 20
  3. 2025Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generationcited by 19
  4. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  5. 2025Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generationcited by 17
  6. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  7. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  8. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  9. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  10. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  11. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  12. 2025OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilitiescited by 9
  13. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  14. 2025VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generationcited by 6
  15. 2026Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Modelscited by 6
  16. 2026OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generationcited by 5
  17. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 5
  18. 2025Autoregressive Styled Text Image Generation, but Make it Reliablecited by 3
  19. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  20. 2025Image Diffusion Preview with Consistency Solvercited by 3
  21. 2026WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modelingcited by 3
  22. 2025NeoBabel: A Multilingual Open Tower for Visual Generationcited by 2
  23. 2026HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understandingcited by 2
  24. 2026HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizerscited by 2
  25. 2025Generative Distribution Distillationcited by 1
  26. 2026Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Modelscited by 1
  27. 2026Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Modelscited by 1
  28. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  29. 2025Fake it till You Make it: Reward Modeling as Discriminative Predictioncited by 0
  30. 2025Unison: A Fully Automatic, Task-Universal, and Low-Cost Framework for Unified Understanding and Generationcited by 0
  31. 2026On the Adversarial Robustness of Discrete Image Tokenizerscited by 0
  32. 2026FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoningcited by 0
  33. 2026Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Modelcited by 0
  34. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generationcited by 0
  35. 2026Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applicationscited by 0
  36. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspectcited by 0
  37. 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World Statescited by 0
  38. 2026Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Modelscited by 0

50 most cited of 51 All 51 (JSON)