SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have achieved remarkable success in visual understanding but remain constrained in visual generation due to the fundamental feature discrepancy between semantic perception and pixel-level reconstruction. Bridging this gap requires overcoming two core challenges: endowing semantic encoders with high-fidelity reconstruction capabilities, and effectively aligning generative models with semantic spaces without relying on external teachers. To this end, we propose a novel unified multimodal framework featuring Semantic-Pixel self-alignment and Adaptive Routing (SPAR). First, to reconcile semantic perception with pixel-level reconstruction, we introduce an asymmetric dual-stream unified tokenizer. A lightweight semantic stream anchors discriminative features, while a Transformer-augmented pixel stream recovers fine-grained visual details into a unified compact latent space. Second, to eliminate external dependencies, we propose a self-aligned generation paradigm that natively leverages this optimized tokenizer as an internal alignment teacher for the diffusion model. Furthermore, to facilitate flexible multimodal interaction within this unified space, we introduce Dynamic Token Routing, which enables each token to adaptively aggregate multi-layer MLLM features based on its distinct semantic demands. Extensive experiments demonstrate that SPAR establishes the state-of-the-art for unified architectures, achieving exceptional generation and reconstruction quality while preserving foundational visual understanding capabilities.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 83 references that are in the map. First posted on arXiv 22 Jun 2026.
Builds on
75 of 83 references.
Closest ancestors · 10
Same direction, newest first
- 2025Qwen3-VL Technical Report26 Nov
- 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations23 Jun
- 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
- 2023Generative Multimodal Models are In-Context Learners20 Dec
- 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
- 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
- 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
Other references · 65
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2025Qwen3 Technical Reportcited by 8,820
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Visual Instruction Tuningcited by 7,869
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2017Neural Discrete Representation Learningcited by 4,712
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
38 more
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
- 2025Qwen-Image Technical Reportcited by 1,044
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2024OmniGen: Unified Image Generationcited by 394
- 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
- 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
- 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
- 2025Diffusion Transformers with Representation Autoencoderscited by 295
- 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
- 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
- 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 217
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
- 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
- 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
- 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
- 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
- 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
- 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 97
- 2025Latent Diffusion Model without Variational Autoencodercited by 88
- 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
- 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
- 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
- 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
- 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
- 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
- 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
- 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
- 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models