MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Unified multimodal Large Language Models (LLMs) that can both understand and generate visual content hold immense potential. However, existing open-source models often suffer from a performance trade-off between these capabilities. We present Manzano, a simple and scalable unified framework that substantially reduces this tension by coupling a hybrid image tokenizer with a well-curated training recipe. A single shared vision encoder feeds two lightweight adapters that produce continuous embeddings for image-to-text understanding and discrete tokens for text-to-image generation within a common semantic space. A unified autoregressive LLM predicts high-level semantics in the form of text and image tokens, with an auxiliary diffusion decoder subsequently translating the image tokens into pixels. The architecture, together with a unified training recipe over understanding and generation data, enables scalable joint learning of both capabilities. Manzano achieves state-of-the-art results among unified models, and is competitive with specialist models, particularly on text-rich evaluation. Our studies show minimal task conflicts and consistent gains from scaling model size, validating our design choice of a hybrid tokenizer.
Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 82 of its 91 references that are in the map. First posted on arXiv 19 Sep 2025.
Builds on
82 of 91 references.
Closest ancestors · 15
Same direction, newest first
- 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations23 Jun
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
- 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
- 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models16 Aug
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
- 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
- 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
- 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
- 2023VILA: On Pre-training for Visual Language Models12 Dec
- 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
3 more
Other references · 67
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
- 2023Visual Instruction Tuningcited by 7,869
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Flow Matching for Generative Modelingcited by 5,702
38 more
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2017Neural Discrete Representation Learningcited by 4,712
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2019Towards VQA Models That Can Readcited by 1,718
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
- 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
- 2025Qwen-Image Technical Reportcited by 1,044
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
- 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
- 2016A Diagram Is Worth A Dozen Imagescited by 804
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
- 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
- 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
50 most cited of 67 All 67 (JSON)
Built on by
Citing papers per year: 2025: 4; 2026: 11.
15 papers.
Sort:
By area:
- 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
- 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
- 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 10
- 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
- 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 8
- 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
- 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
- 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
- 2026Adaptive 1D Video Diffusion Autoencodercited by 3
- 2026CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructionscited by 3
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 3
- 2026Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generationcited by 2
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes5 Aug 2026
- 2026Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation25 Jun 2026
- 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens18 May 2026
- 2026Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learning11 May 2026
- 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model2 Apr 2026
- 2026CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions27 Mar 2026
- 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining3 Mar 2026
- 2026The Design Space of Tri-Modal Masked Diffusion Models25 Feb 2026
- 2026Kelix Technical Report10 Feb 2026
- 2026Adaptive 1D Video Diffusion Autoencoder4 Feb 2026
- 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Code23 Jan 2026
- 2025TV2TV: A Unified Framework for Interleaved Language and Video Generation4 Dec 2025
3 more
- 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
- 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
- 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
- 2026Adaptive 1D Video Diffusion Autoencodercited by 3
- 2026The Design Space of Tri-Modal Masked Diffusion Models25 Feb 2026
- 2026Adaptive 1D Video Diffusion Autoencoder4 Feb 2026
- 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Code23 Jan 2026
- 2025BLIP3o-NEXT: Next Frontier of Native Image Generation17 Oct 2025
- 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 10
- 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Modelcited by 8
- 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
- 2026CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructionscited by 3
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 3
- 2026Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generationcited by 2
- 2026Kelix Technical Reportcited by 1
- 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
- 2026Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learningcited by 0
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes5 Aug 2026
- 2026Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation25 Jun 2026
- 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens18 May 2026
- 2026Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learning11 May 2026
- 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model2 Apr 2026
- 2026CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions27 Mar 2026
- 2026Kelix Technical Report10 Feb 2026
- 2025TV2TV: A Unified Framework for Interleaved Language and Video Generation4 Dec 2025
- 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward25 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2023Planting a SEED of Vision in Large Language Model
- 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
- 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
- 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
- 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm
- 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
- 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation
- 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability