HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization
Not on the map. Why?
Abstract (arXiv)
Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of 10.0 points across eight challenging understanding benchmarks.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 70 of its 74 references that are in the map. First posted on arXiv 16 Mar 2026.
Builds on
70 of 74 references.
References · 70
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2023Segment Anythingcited by 8,158
- 2023Visual Instruction Tuningcited by 7,792
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2023Qwen Technical Reportcited by 3,273
38 more
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
- 2019An Introduction to Variational Autoencoderscited by 997
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
- 2023DataComp: In search of the next generation of multimodal datasetscited by 635
- 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2024The Platonic Representation Hypothesiscited by 362
- 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
- 2025Transfer between Modalities with MetaQueriescited by 231
- 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
- 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
- 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
- 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
- 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.