# Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models

arXiv: https://arxiv.org/abs/2511.19917 · HTML: https://mlsanity.com/paper/2511.19917/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning.
- Builds on 22 of its 68 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- LLM reasoning / [Diffusion reasoning and test-time scaling](https://mlsanity.com/lr/diffusion-reasoning-and-test-time-scaling/)
- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Stochastic Interpolants: A Unifying Framework for Flows and Diffusions](https://mlsanity.com/paper/2303.08797/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [A General Framework for Inference-time Scaling and Steering of Diffusion Models](https://mlsanity.com/paper/2501.06848/) (2025)
- [Test-time Alignment of Diffusion Models without Reward Over-optimization](https://mlsanity.com/paper/2501.05803/) (2025)
- [Progressive Distillation for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2202.00512/) (2022)
- [ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features](https://mlsanity.com/paper/2502.04320/) (2025)
- [Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.00637/) (2023)
- [Scaling Image and Video Generation via Test-Time Evolutionary Search](https://mlsanity.com/paper/2505.17618/) (2025)
- [Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing](https://mlsanity.com/paper/2503.19385/) (2025)
- [Test-Time Scaling of Diffusion Models via Noise Trajectory Search](https://mlsanity.com/paper/2506.03164/) (2025)
- [CoMA: Compositional Human Motion Generation with Multi-modal Agents](https://mlsanity.com/paper/2412.07320/) (2024)
- [Stochastic Sampling from Deterministic Flow Models](https://mlsanity.com/paper/2410.02217/) (2024)
- [Exploring Diffusion and Flow Matching Under Generator Matching](https://mlsanity.com/paper/2412.11024/) (2024)
- [Fast Monte Carlo Tree Diffusion: 100x Speedup via Parallel Sparse Planning](https://mlsanity.com/paper/2506.09498/) (2025)
- [Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis](https://mlsanity.com/paper/2511.18089/) (2025)

## Built on by (most cited first, 6 of 6)

- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)
- [NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction](https://mlsanity.com/paper/2606.24087/) (2026)
- [Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models](https://mlsanity.com/paper/2512.20954/) (2025)
- [Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation](https://mlsanity.com/paper/2605.10009/) (2026)
- [FreeBridge: Variational Schr"odinger Bridges for Cellular Transition Dynamics](https://mlsanity.com/paper/2606.11286/) (2026)
- [Test-Time Scaling for Video Diffusion Models via Diagnosis-Guided Candidate Recycling](https://mlsanity.com/paper/2608.29322/) (2026)

## Abstract (arXiv)

Diffusion models have become the dominant paradigm in text-to-image generation, and test-time scaling (TTS) improves sample quality by allocating additional computation at inference. Existing TTS methods, however, resample the entire image, while generation quality is often spatially heterogeneous. This leads to unnecessary computation on regions that are already correct, and localized defects remain insufficiently corrected. In this paper, we explore a new direction - Localized TTS - that adaptively resamples defective regions while preserving high-quality regions, thereby substantially reducing the search space. This raises two challenges: accurately localizing defects and maintaining global consistency. We propose LoTTS, the first fully training-free framework for localized TTS. For defect localization, LoTTS contrasts cross- and self-attention signals under quality-aware prompts (e.g., high-quality vs. low-quality) to identify defective regions, and then refines them into coherent masks. For consistency, LoTTS perturbs only defective regions and denoises them locally, ensuring that corrections remain confined while the rest of the image remains undisturbed. Extensive experiments on SD2.1, SDXL, and FLUX demonstrate that LoTTS achieves state-of-the-art performance: it consistently improves both local quality and global fidelity, while reducing GPU cost by 2-4x compared to Best-of-N sampling. These findings establish localized TTS as a promising new direction for scaling diffusion models at inference time.