mlsanityData to 7 Oct 2026

FiT: Flexible Vision Transformer for Diffusion Model

arXiv 2402.12376 ↗First posted 19 Feb 2024cs.CVcited by 84

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Nature is infinitely resolution-free. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To overcome this limitation, we present the Flexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with unrestricted resolutions and aspect ratios. Unlike traditional methods that perceive images as static-resolution grids, FiT conceptualizes images as sequences of dynamically-sized tokens. This perspective enables a flexible training strategy that effortlessly adapts to diverse aspect ratios during both training and inference phases, thus promoting resolution generalization and eliminating biases induced by image cropping. Enhanced by a meticulously adjusted network structure and the integration of training-free extrapolation techniques, FiT exhibits remarkable flexibility in resolution extrapolation generation. Comprehensive experiments demonstrate the exceptional performance of FiT across a broad range of resolutions, showcasing its effectiveness both within and beyond its training resolution distribution. Repository available at https://github.com/whlzy/FiT.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

38 of 45 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023DiffiT: Diffusion Vision Transformers for Image Generation4 Dec
  2. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  3. 2022Scalable Diffusion Models with Transformers19 Dec
  4. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  5. 2022Classifier-Free Diffusion Guidance26 Jul
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  8. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  10. 2020Denoising Diffusion Implicit Models6 Oct
  11. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2022A ConvNet for the 2020scited by 3,545
15 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  6. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  9. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  10. 2021Generating Images with Sparse Representationscited by 262
  11. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  12. 2019Fixing the train-test resolution discrepancycited by 209
  13. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  14. 2022A Length-Extrapolatable Transformercited by 123
  15. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 37; 2026: 14.

84 papers.

Sort:

By area:

  1. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  2. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
  3. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  4. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  5. 2024Rotary Position Embedding for Vision Transformercited by 226
  6. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  7. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  8. 2025DDT: Decoupled Diffusion Transformercited by 100
  9. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  10. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  11. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 63
  12. 2024FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesiscited by 58
38 more
  1. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 55
  2. 2024U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformerscited by 51
  3. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  4. 2024Scaling Laws For Diffusion Transformerscited by 41
  5. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  6. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  7. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  8. 2025U-REPA: Aligning Diffusion U-Nets to ViTscited by 29
  9. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28
  10. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  11. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  12. 2024ResMaster: Mastering High-Resolution Image Generation via Structural and Fine-Grained Guidancecited by 21
  13. 2024I-Max: Maximize the Resolution Potential of Pre-trained Rectified Flow Transformers with Projected Flowcited by 21
  14. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Drivingcited by 21
  15. 2024Effective Diffusion Transformer Architecture for Image Super-Resolutioncited by 20
  16. 2024TVG: A Training-free Transition Video Generation Method with Diffusion Modelscited by 15
  17. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  18. 2024FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolutioncited by 15
  19. 2024HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Promptscited by 14
  20. 2025MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTscited by 14
  21. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  22. 2025Native-Resolution Image Synthesiscited by 13
  23. 2025DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusioncited by 13
  24. 2024Diffusion Models Need Visual Priors for Image Generationcited by 12
  25. 2025Vision Transformers with Self-Distilled Registerscited by 12
  26. 2025UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformerscited by 10
  27. 2024Neural Residual Diffusion Models for Deep Scalable Vision Generationcited by 9
  28. 2024MVOC: a training-free multiple video object composition method with diffusion modelscited by 8
  29. 2024EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Modelscited by 8
  30. 2025D2iT: Dynamic Diffusion Transformer for Accurate Image Generationcited by 8
  31. 2025FLEX: A Backbone for Diffusion-Based Modeling of Spatio-temporal Physical Systemscited by 8
  32. 2025Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivotscited by 8
  33. 2025HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Modelscited by 8
  34. 2025PanoSplatt3R: Leveraging Perspective Pretraining for Generalized Unposed Wide-Baseline Panorama Reconstructioncited by 7
  35. 2025Understand Before You Generate: Self-Guided Training for Autoregressive Image Generationcited by 7
  36. 2025Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Boundscited by 7
  37. 2024PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplinescited by 6
  38. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance
  2. 2026One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformers
  3. 2024Diffscaler: Enhancing the Generative Prowess of Diffusion Transformers
  4. 2026Pixel-Space Diffusion Transformers
  5. 2024LaVin-DiT: Large Vision Diffusion Transformer
  6. 2025DiP: Taming Diffusion Models in Pixel Space
  7. 2026HRDiT: Training-Free High-Resolution Image Generation with Off-the-Shelf Diffusion Transformer Models
  8. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models