mlsanityData to 7 Oct 2026

DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space

arXiv 2412.15032 ↗First posted 19 Dec 2024cs.CV cs.LG eess.IVcited by 32

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to 512×512 resolution without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why 'image diffusion can be seen as spectral autoregression', bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is https://github.com/forever208/DCTdiff.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 59 of its 79 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

59 of 79 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Improved Noise Schedule for Diffusion Training3 Jul
  2. 2024Align Your Steps: Optimizing Sampling Schedules in Diffusion Models22 Apr
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  4. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  5. 2023Fast ODE-based Sampling for Diffusion Models in Around 5 Steps30 Nov
  6. 2023SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models10 Sep
  7. 2023Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps24 May
  8. 2023Consistency Models2 Mar
  9. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation1 Mar
  10. 2023Input Perturbation Reduces Exposure Bias in Diffusion Models27 Jan
  11. 2023Simple diffusion: End-to-end diffusion for high resolution images26 Jan
  12. 2022Scalable Diffusion Models with Transformers19 Dec
12 more
  1. 2022Wavelet Diffusion Models are fast and scalable Image Generators29 Nov
  2. 2022Diffusion Probabilistic Model Made Slim27 Nov
  3. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
  4. 2022Classifier-Free Diffusion Guidance26 Jul
  5. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  7. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models17 Jan
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  9. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  11. 2020Denoising Diffusion Implicit Models6 Oct
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
23 more
  1. 2016Improved Techniques for Training GANscited by 2,938
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  4. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
  5. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  6. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  7. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  8. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  9. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  10. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  11. 2021Generating Images with Sparse Representationscited by 262
  12. 2022Poisson Flow Generative Modelscited by 95
  13. 2013Fast Training of Convolutional Networks through FFTscited by 89
  14. 2023Adaptive Frequency Filters As Efficient Global Token Mixerscited by 45
  15. 2020Language Through a Prism: A Spectral Approach for Multiscale Language Representationscited by 39
  16. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 36
  17. 2024Spectral Image Tokenizercited by 15
  18. 2021Fourier Image Transformercited by 12
  19. 2018Learning Long Term Dependencies via Fourier Recurrent Unitscited by 11
  20. 2024Wavelet-Based Image Tokenizer for Vision Transformerscited by 10
  21. 2024Wavelets Are All You Need for Autoregressive Image Generationcited by 9
  22. 2024Hierarchical VAE with a Diffusion-based VampPriorcited by 8
  23. 2024JPEG-LM: LLMs as Image Generators with Canonical Codec Representationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 20.

32 papers.

Sort:

By area:

  1. 2025Improving the Diffusability of Autoencoderscited by 74
  2. 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 43
  3. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 21
  4. 2025SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learningcited by 17
  5. 2025Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusioncited by 16
  6. 2025Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representationcited by 12
  7. 2026Unleashing the Potential of Diffusion Models for End-to-End Autonomous Drivingcited by 11
  8. 2025FreqPolicy: Frequency Autoregressive Visuomotor Policy with Continuous Tokenscited by 9
  9. 2025POLARIS: Projection-Orthogonal Least Squares for Robust and Adaptive Inversion in Diffusion Modelscited by 8
  10. 2025Improving Progressive Generation with Decomposable Flow Matchingcited by 6
  11. 2026Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusioncited by 6
  12. 2025Free-T2M: Robust Text-to-Motion Generation for Humanoid Robots via Frequency-Domaincited by 5
20 more
  1. 2026ECHO: Edge-Cloud Humanoid Orchestration for Language-to-Motion Controlcited by 4
  2. 2026SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matchingcited by 4
  3. 2026Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translationcited by 4
  4. 2025WaveletDiff: Multilevel Wavelet Diffusion For Time Series Generationcited by 2
  5. 2026Oracle Noise: Faster Semantic Spherical Alignment for Interpretable Latent Optimizationcited by 2
  6. 2026Functional MRI Time Series Generation via Wavelet-Based Image Transform and Spectral Flow Matching for Brain Disorder Identificationcited by 2
  7. 2026MemoGen: Can Past Experience Improve Future Text-to-Image Generation?cited by 2
  8. 2026Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusioncited by 2
  9. 2026WaiT for the Signal: Simple Frequency-Aware Flow-Matchingcited by 2
  10. 2025The Mechanics of CNN Filtering with Rectificationcited by 1
  11. 2026Z2-Sampling: Zero-Cost Zigzag Trajectories for Semantic Alignment in Diffusion Modelscited by 1
  12. 2026Everything at Every Scale: Scale-Invariant Diffusion with Continuous Super-Resolutioncited by 1
  13. 2026Cyclostationary Phase Conditioning for Medical Time Series Diffusioncited by 1
  14. 2026Dynamic Frequency Modulation for Controllable Text-driven Image Generationcited by 0
  15. 2026Structured State-Space Regularization for Generation-Friendly Image Tokenizationcited by 0
  16. 2026Before the Body Moves: Learning Anticipatory Joint Intent for Language-Conditioned Humanoid Controlcited by 0
  17. 2026Learning to Think in Physics: Breaking Shortcut Learning in Scientific Diffusion via Representation Alignmentcited by 0
  18. 2026HyFAD: Hybrid Time-Frequency Diffusion with Frequency-Aware Embedding for Time Series Imputationcited by 0
  19. 2026Fourier-Latent Diffusion for Constrained Generation of Triply Periodic Minimal Surfacescited by 0
  20. 2026Beyond the Manifold Hypothesis: Hybrid Spectral Parameterizations for Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition
  2. 2023DiffiT: Diffusion Vision Transformers for Image Generation
  3. 2025ReDDiT: Rehashing Noise for Discrete Visual Generation
  4. 2022Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image Synthesis
  5. 2023Diffusion Models Without Attention
  6. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion
  7. 2025DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusion
  8. 2024DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents