# Fourier Preconditioning for Neural Feature Learning

arXiv: https://arxiv.org/abs/2607.02199 · HTML: https://mlsanity.com/paper/2607.02199/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 9 of 9)

- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Stationary Graph Processes and Spectral Estimation](https://mlsanity.com/paper/1603.04667/) (2016)
- [Neural Feature Learning in Function Space](https://mlsanity.com/paper/2309.10140/) (2023)
- [Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment](https://mlsanity.com/paper/2106.14699/) (2021)
- [Mutual Information in Frequency and its Application to Measure Cross-Frequency Coupling in Epilepsy](https://mlsanity.com/paper/1711.01629/) (2017)
- [Mutual Information-Driven Visualization and Clustering for Core KPI Selection in O-RAN Testing](https://mlsanity.com/paper/2510.02696/) (2025)
- [Learning Low-Dimensional Representation for O-RAN Testing via Transformer-ESN](https://mlsanity.com/paper/2604.12958/) (2026)

## Abstract (arXiv)

Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental.