# RefAlign: Representation Alignment for Reference-to-Video Generation arXiv: https://arxiv.org/abs/2603.25743 ยท HTML: https://mlsanity.com/paper/2603.25743/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 53 references that are in the map. - First posted on arXiv 26 Mar 2026. ## Builds on (most cited first, 30 of 48) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024) - [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025) - [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024) - [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024) - [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025) - [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025) - [REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers](https://mlsanity.com/paper/2504.10483/) (2025) - [Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models](https://mlsanity.com/paper/2405.04233/) (2024) - [HunyuanVideo 1.5 Technical Report](https://mlsanity.com/paper/2511.18870/) (2025) - [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://mlsanity.com/paper/2411.17440/) (2024) - [Phantom: Subject-consistent video generation via cross-modal alignment](https://mlsanity.com/paper/2502.11079/) (2025) - [DDT: Decoupled Diffusion Transformer](https://mlsanity.com/paper/2504.05741/) (2025) - [Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think](https://mlsanity.com/paper/2507.01467/) (2025) - [VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models](https://mlsanity.com/paper/2505.23656/) (2025) - [HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation](https://mlsanity.com/paper/2505.04512/) (2025) - [SkyReels-A2: Compose Anything in Video Diffusion Transformers](https://mlsanity.com/paper/2504.02436/) (2025) - [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025) - [Kling-Omni Technical Report](https://mlsanity.com/paper/2512.16776/) (2025) ## Built on by (most cited first, 10 of 10) - [Bernini: Latent Semantic Planning for Video Diffusion](https://mlsanity.com/paper/2605.22344/) (2026) - [VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing](https://mlsanity.com/paper/2608.05485/) (2026) - [Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping](https://mlsanity.com/paper/2605.10937/) (2026) - [RECAP-Forcing: Retaining Content Appearances for Long Video Generation](https://mlsanity.com/paper/2608.26671/) (2026) - [Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment](https://mlsanity.com/paper/2607.04311/) (2026) - [HarmoView: Harmonizing Multi-View Constraints for Identity-Consistent Video Generation](https://mlsanity.com/paper/2606.10839/) (2026) - [HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement](https://mlsanity.com/paper/2607.18217/) (2026) - [SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models](https://mlsanity.com/paper/2605.07800/) (2026) - [KeyID: Decoupled Drafting and Keyframe Editing for Identity-Preserving Video Generation](https://mlsanity.com/paper/2608.16154/) (2026) - [AdSpark: A Large-Scale Dataset and Benchmark for Product-Centric Advertisement Video Generation](https://mlsanity.com/paper/2610.10047/) (2026) ## Abstract (arXiv) Reference-to-video (R2V) generation is a controllable video synthesis paradigm that constrains the generation process using both text prompts and reference images, enabling applications such as personalized advertising and virtual try-on. In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image and jointly feed them into the diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy--paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoder features. In this paper, we propose RefAlign, a representation alignment framework that explicitly aligns DiT reference-branch features to the semantic space of a visual foundation model (VFM). The core of RefAlign is a reference alignment loss that pulls the reference features and VFM features of the same subject closer to improve identity consistency, while pushing apart the corresponding features of different subjects to enhance semantic discriminability. This simple yet effective strategy is applied only during training, incurring no inference-time overhead, and achieves a better balance between text controllability and reference fidelity. Extensive experiments on the OpenS2V-Eval benchmark demonstrate that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference alignment for R2V tasks.