When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities
On the map: Diffusion models → Diffusion foundations and sampling
Abstract (arXiv)
Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers dominated by linear operations. Analog compute-in-memory (CIM) can alleviate these costs by executing linear operations within weight-storing memory arrays. However, CIM nonidealities perturb effective weights, with errors accumulating along the state-dependent denoising trajectory; their interaction with classifier-free guidance (CFG) remains underexplored. In this paper, we characterize the impact of analog CIM nonidealities on DiT sampling. Although conditional and unconditional predictions can each remain close to their clean counterparts, their difference (the CFG residual) is disproportionately attenuated and rotated. Identifying this residual as a controllable failure channel, we propose a retraining-free, sampler-side recalibration that adjusts only the CFG scale for a given CIM condition. Trajectory-level analysis shows that moderate recalibration strengthens the target-oriented component preserved in the distorted residual, enabling earlier commitment to a prompt-consistent semantic region. In contrast, excessive guidance amplifies the full noisy residual and degrades quality, resulting in a finite, noise-dependent optimum. Extensive experiments on PixArt-Sigma, PixArt-alpha, and DiT-XL/2 show that the optimal guidance scale increases with CIM noise. Using 30,000 samples per condition, guidance recalibration consistently restores generation quality across simulated CIM mappings, closing at least 87% of the CIM-induced FID gap at a CIM noise level of 0.20. It reduces FID from 59.22 to 20.49 on PixArt-Sigma, 72.37 to 21.12 on PixArt-alpha, and 20.89 to 6.62 on DiT-XL/2.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 19 of its 32 references that are in the map. First posted on arXiv 20 Aug 2026.
Builds on
19 of 32 references.
Closest ancestors · 10
Same direction, newest first
- 2024Classifier-Free Guidance inside the Attraction Basin May Cause Memorization23 Nov
- 2024Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models3 Oct
- 2024Classifier-Free Guidance is a Predictor-Corrector16 Aug
- 2024CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models12 Jun
- 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models11 Apr
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
- 2023Common Diffusion Noise Schedules and Sample Steps are Flawed15 May
- 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
- 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
- 2020Denoising Diffusion Probabilistic Models19 Jun
Other references · 9
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
- 2014Microsoft COCO: Common Objects in Contextcited by 17,550
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,407
- 2018Demystifying MMD GANscited by 1,277
- 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 928
- 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 428
- 2025Leveraging Compute-in-Memory for Efficient Generative Model Inference in TPUscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware
- 2025RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer
- 2025Exploring Magnitude Preservation and Rotation Modulation in Diffusion Transformers
- 2026Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache
- 2026BeCARE: Budgeted Cache Refresh for Diffusion Transformer Acceleration
- 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
- 2025FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers
- 2025CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformers