# Representation Alignment for Just Image Transformers is not Easier than You Think

arXiv: https://arxiv.org/abs/2603.14366 · HTML: https://mlsanity.com/paper/2603.14366/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation](https://mlsanity.com/paper/2303.00848/) (2023)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers](https://mlsanity.com/paper/2504.10483/) (2025)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Scalable Adaptive Computation for Iterative Generation](https://mlsanity.com/paper/2212.11972/) (2022)
- [JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2411.07975/) (2024)
- [DDT: Decoupled Diffusion Transformer](https://mlsanity.com/paper/2504.05741/) (2025)

## Built on by (most cited first, 14 of 14)

- [Asymmetric Flow Models](https://mlsanity.com/paper/2605.12964/) (2026)
- [Registers Matter for Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2605.16147/) (2026)
- [Persistence Forcing: Exploiting Feature Specialization in Pixel-Space Diffusion](https://mlsanity.com/paper/2609.36014/) (2026)
- [GEAR: Guided End-to-End AutoRegression for Image Synthesis](https://mlsanity.com/paper/2606.32039/) (2026)
- [Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2607.17585/) (2026)
- [FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders](https://mlsanity.com/paper/2609.31620/) (2026)
- [RepFlow: Reciprocal Supervision Improves Generation and Representation in Flow Models](https://mlsanity.com/paper/2609.33217/) (2026)
- [Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers](https://mlsanity.com/paper/2609.33895/) (2026)
- [Scaffold Then Internalize: Representation Injection for Diffusion Transformers](https://mlsanity.com/paper/2609.35292/) (2026)
- [PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion](https://mlsanity.com/paper/2610.00483/) (2026)
- [Just Align bmx: Aligning Predictions, Not Representations](https://mlsanity.com/paper/2610.00600/) (2026)
- [TAME:Topology-Aware Text-Driven Motion Editing across Heterogeneous Humanoid Skeletons](https://mlsanity.com/paper/2610.04529/) (2026)
- [Learning Representations to Guide Pixel Diffusion Transformers](https://mlsanity.com/paper/2610.12978/) (2026)
- [PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts](https://mlsanity.com/paper/2610.18857/) (2026)

## Abstract (arXiv)

Representation Alignment (REPA) has emerged as a simple way to accelerate Diffusion Transformers training in latent space. At the same time, pixel-space diffusion transformers such as Just image Transformers (JiT) have attracted growing attention because they remove a dependency on a pretrained tokenizer, and then avoid the reconstruction bottleneck of latent diffusion. This paper shows that the REPA can fail for JiT. REPA yields worse FID for JiT as training proceeds and collapses diversity on image subsets that are tightly clustered in the representation space of pretrained semantic encoder on ImageNet. We trace the failure to an information asymmetry: denoising occurs in the high dimensional image space, while the semantic target is strongly compressed, making direct regression a shortcut objective. We propose PixelREPA, which transforms the alignment target and constrains alignment with a Masked Transformer Adapter that combines a shallow transformer adapter with partial token masking. PixelREPA improves both training convergence and final quality. PixelREPA reduces FID from 3.66 to 3.17 for JiT-B/16 and improves Inception Score (IS) from 275.1 to 284.6 on ImageNet 256 × 256, while achieving > 2× faster convergence. Finally, PixelREPA-H/16 achieves FID=1.81 and IS=317.2. Our code is available at https://github.com/kaist-cvml/PixelREPA.