mlsanityData to 7 Oct 2026

HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization

arXiv 2603.15228 ↗First posted 16 Mar 2026cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of 10.0 points across eight challenging understanding benchmarks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 70 of its 74 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

70 of 74 references.

References · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  11. 2019An Introduction to Variational Autoencoderscited by 997
  12. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  13. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  15. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  16. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  17. 2016A Diagram Is Worth A Dozen Imagescited by 793
  18. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  19. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  20. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  21. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  22. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  23. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  24. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  25. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  26. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  27. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  28. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  29. 2024The Platonic Representation Hypothesiscited by 362
  30. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  31. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  32. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  33. 2025Transfer between Modalities with MetaQueriescited by 231
  34. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  35. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  36. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  37. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  38. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.