# ToDo: Token Downsampling for Efficient Generation of High-Resolution Images

arXiv: https://arxiv.org/abs/2402.13573 · HTML: https://mlsanity.com/paper/2402.13573/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 13 of 13)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [A Survey on Efficient Training of Transformers](https://mlsanity.com/paper/2302.01107/) (2023)
- [AReLU: Attention-based Rectified Linear Unit](https://mlsanity.com/paper/2006.13858/) (2020)

## Built on by (most cited first, 14 of 14)

- [Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT](https://mlsanity.com/paper/2406.18583/) (2024)
- [U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers](https://mlsanity.com/paper/2405.02730/) (2024)
- [Importance-Based Token Merging for Efficient Image and Video Generation](https://mlsanity.com/paper/2411.16720/) (2024)
- [Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers](https://mlsanity.com/paper/2412.16822/) (2024)
- [DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers](https://mlsanity.com/paper/2503.22796/) (2025)
- [RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy](https://mlsanity.com/paper/2505.21036/) (2025)
- [Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment](https://mlsanity.com/paper/2508.06160/) (2025)
- [DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.16968/) (2026)
- [CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers](https://mlsanity.com/paper/2605.14191/) (2026)
- [BiGain: Unified Token Compression for Joint Generation and Classification](https://mlsanity.com/paper/2603.12240/) (2026)
- [Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness](https://mlsanity.com/paper/2605.22011/) (2026)
- [HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing](https://mlsanity.com/paper/2606.13898/) (2026)
- [Tuning-Free Latent Diffusion Models for Ultrahigh-Resolution Image Editing](https://mlsanity.com/paper/2607.06136/) (2026)
- [Importance-Aware OBS Pruning for Diffusion Models](https://mlsanity.com/paper/2607.20048/) (2026)

## Abstract (arXiv)

Attention mechanism has been crucial for image diffusion models, however, their quadratic computational complexity limits the sizes of images we can process within reasonable time and memory constraints. This paper investigates the importance of dense attention in generative image models, which often contain redundant features, making them suitable for sparser attention mechanisms. We propose a novel training-free method ToDo that relies on token downsampling of key and value tokens to accelerate Stable Diffusion inference by up to 2x for common sizes and up to 4.5x or more for high resolutions like 2048x2048. We demonstrate that our approach outperforms previous methods in balancing efficient throughput and fidelity.