mlsanityData to 14 Oct 2026

SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models

arXiv 2606.23041 ↗First posted 22 Jun 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved remarkable success in visual understanding but remain constrained in visual generation due to the fundamental feature discrepancy between semantic perception and pixel-level reconstruction. Bridging this gap requires overcoming two core challenges: endowing semantic encoders with high-fidelity reconstruction capabilities, and effectively aligning generative models with semantic spaces without relying on external teachers. To this end, we propose a novel unified multimodal framework featuring Semantic-Pixel self-alignment and Adaptive Routing (SPAR). First, to reconcile semantic perception with pixel-level reconstruction, we introduce an asymmetric dual-stream unified tokenizer. A lightweight semantic stream anchors discriminative features, while a Transformer-augmented pixel stream recovers fine-grained visual details into a unified compact latent space. Second, to eliminate external dependencies, we propose a self-aligned generation paradigm that natively leverages this optimized tokenizer as an internal alignment teacher for the diffusion model. Furthermore, to facilitate flexible multimodal interaction within this unified space, we introduce Dynamic Token Routing, which enables each token to adaptively aggregate multi-layer MLLM features based on its distinct semantic demands. Extensive experiments demonstrate that SPAR establishes the state-of-the-art for unified architectures, achieving exceptional generation and reconstruction quality while preserving foundational visual understanding capabilities.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 83 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

75 of 83 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations23 Jun
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  5. 2023Generative Multimodal Models are In-Context Learners20 Dec
  6. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 65

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
38 more
  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  3. 2025Qwen-Image Technical Reportcited by 1,044
  4. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  5. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  6. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  9. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  12. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  13. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  14. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  15. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  16. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  17. 2024OmniGen: Unified Image Generationcited by 394
  18. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  19. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  20. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  21. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  22. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  23. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  24. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  25. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  26. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  27. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  28. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 217
  29. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  30. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  31. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
  32. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  33. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  34. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  35. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 97
  36. 2025Latent Diffusion Model without Variational Autoencodercited by 88
  37. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  38. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  2. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  3. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  4. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  5. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
  6. 2025SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
  7. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  8. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models