# Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise

arXiv: https://arxiv.org/abs/2510.09660 · HTML: https://mlsanity.com/paper/2510.09660/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://mlsanity.com/paper/2208.09392/) (2022)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature](https://mlsanity.com/paper/1505.00855/) (2015)
- [On the Inductive Bias of Neural Tangent Kernels](https://mlsanity.com/paper/1905.12173/) (2019)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Generative Modelling With Inverse Heat Dissipation](https://mlsanity.com/paper/2206.13397/) (2022)
- [Amortizing intractable inference in diffusion models for vision, language, and control](https://mlsanity.com/paper/2405.20971/) (2024)
- [Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective](https://mlsanity.com/paper/2110.03095/) (2021)
- [Boosting Diffusion Models with Moving Average Sampling in Frequency Domain](https://mlsanity.com/paper/2403.17870/) (2024)
- [Diffusion Models With Learned Adaptive Noise](https://mlsanity.com/paper/2312.13236/) (2023)
- [Time Series Diffusion in the Frequency Domain](https://mlsanity.com/paper/2402.05933/) (2024)
- [Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases](https://mlsanity.com/paper/2402.08552/) (2024)
- [Blue noise for diffusion models](https://mlsanity.com/paper/2402.04930/) (2024)
- [Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models](https://mlsanity.com/paper/2210.12254/) (2022)
- [Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2307.14648/) (2023)
- [Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks](https://mlsanity.com/paper/2310.02230/) (2023)
- [FDG-Diff: Frequency-Domain-Guided Diffusion Framework for Compressed Hazy Image Restoration](https://mlsanity.com/paper/2501.12832/) (2025)

## Built on by (most cited first, 2 of 2)

- [Colored Noise Diffusion Sampling](https://mlsanity.com/paper/2605.30332/) (2026)
- [PRISM: Principled Reference Identification for Schrodinger Bridge Model](https://mlsanity.com/paper/2608.06893/) (2026)

## Abstract (arXiv)

Diffusion Probabilistic Models (DPMs) have achieved strong generative performance, yet their inductive biases remain largely implicit. In this work, we aim to build inductive biases into the training and sampling of diffusion models to better accommodate the target distribution of the data to model. We introduce an anisotropic noise operator that shapes these biases by replacing the isotropic forward covariance with a structured, frequency-diagonal covariance. This operator unifies band-pass masks and power-law weightings, allowing us to emphasize or suppress designated frequency bands, while keeping the forward process Gaussian. We refer to this as Spectrally Anisotropic Gaussian Diffusion (SAGD). In this work, we derive the score relation for anisotropic forward covariances and show that, under full support, the learned score converges to the true data score as t!→!0, while anisotropy reshapes the probability-flow path from noise to data. Empirically, we show the induced anisotropy outperforms standard diffusion across several vision datasets, and enables selective omission: learning while ignoring known corruptions confined to specific bands. Together, these results demonstrate that carefully designed anisotropic forward noise provides a simple, yet principled, handle to tailor inductive bias in DPMs.