# HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing

arXiv: https://arxiv.org/abs/2606.13898 · HTML: https://mlsanity.com/paper/2606.13898/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving.
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 36)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Seedream 4.0: Toward Next-generation Multimodal Image Generation](https://mlsanity.com/paper/2509.20427/) (2025)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models](https://mlsanity.com/paper/2410.10733/) (2024)
- [Structural Pruning for Diffusion Models](https://mlsanity.com/paper/2305.10924/) (2023)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019)
- [Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching](https://mlsanity.com/paper/2406.01733/) (2024)
- [UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics](https://mlsanity.com/paper/2412.07774/) (2024)
- [Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers](https://mlsanity.com/paper/2406.17343/) (2024)
- [Distilling Diffusion Models into Conditional GANs](https://mlsanity.com/paper/2405.05967/) (2024)
- [Dynamic Diffusion Transformer](https://mlsanity.com/paper/2410.03456/) (2024)
- [DeepCache: Principled Cache for Mobile Deep Vision](https://mlsanity.com/paper/1712.01670/) (2017)
- [Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality](https://mlsanity.com/paper/2505.18227/) (2025)
- [CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training](https://mlsanity.com/paper/2203.11947/) (2022)
- [Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models](https://mlsanity.com/paper/2405.05252/) (2024)
- [Qwen-Image-Layered: Towards Inherent Editability via Layer Decomposition](https://mlsanity.com/paper/2512.15603/) (2025)
- [Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference](https://mlsanity.com/paper/1907.04523/) (2019)
- [Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers](https://mlsanity.com/paper/2412.16822/) (2024)

## Abstract (arXiv)

Creative image editing tools, such as Adobe Photoshop's Remove and Generative Fill buttons, are central to everyday customer use and account for a major share of traffic. However, current generative AI models face significant latency challenges, which become even more pronounced as the field transitions from convolution-based U-Nets to Diffusion Transformers (DiTs). In our evaluation on hundreds of representative image editing samples spanning a wide range of mask ratios, the DiT module alone accounts for an average of 73% of the total model latency, even after being distilled from 50 timesteps down to 8 timesteps. To tackle this challenge, we propose HiLo-Token, an input-adaptive token compression framework that allocates a larger token budget to high-frequency, rich-context regions while assigning fewer tokens to low-frequency areas. Specifically, for the editing region specified by the user mask, we retain all tokens within a dilated mask to preserve strong locality and contextual relevance. Outside the editing region, we introduce a simple yet effective high-frequency token selection strategy based on spatial frequency to capture important local details, while using tokens from a heavily downsampled image to represent low-frequency components and preserve the coarse global structure. Extensive experiments on production-level evaluation data validate the effectiveness of the proposed method, which achieves 3.13×, 2.59×, and 1.67× DiT speedups on A100-80GB for image editing tasks across small, medium, and large mask ratio categories with average ratios of 6.38%, 15.92%, and 35.36%, respectively, without any regression in generation quality. In production, where the editing pipeline is served as disaggregated, independently autoscaled GPU workers, these per-request savings translate into a 33% reduction in the GPUs required to serve the models.