# Learning Rate Should Scale Inversely with High-Order Data Moments in High-Dimensional Online Independent Component Analysis

arXiv: https://arxiv.org/abs/2509.15127 · HTML: https://mlsanity.com/paper/2509.15127/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise](https://mlsanity.com/paper/1906.09069/) (2019)
- [Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions](https://mlsanity.com/paper/2301.11885/) (2023)
- [A Solvable High-Dimensional Model of GAN](https://mlsanity.com/paper/1805.08349/) (2018)
- [Large Dimensional Independent Component Analysis: Statistical Optimality and Computational Tractability](https://mlsanity.com/paper/2303.18156/) (2023)
- [Robust Gaussian Processes via Relevance Pursuit](https://mlsanity.com/paper/2410.24222/) (2024)
- [Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions](https://mlsanity.com/paper/2503.23896/) (2025)
- [Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processes](https://mlsanity.com/paper/1808.09642/) (2018)
- [The Scaling Limit of High-Dimensional Online Independent Component Analysis](https://mlsanity.com/paper/1710.05384/) (2017)
- [Emergence of heavy tails in homogenized stochastic gradient descent](https://mlsanity.com/paper/2402.01382/) (2024)
- [Optimization and Testing in Linear Non-Gaussian Component Analysis](https://mlsanity.com/paper/1712.08837/) (2017)

## Abstract (arXiv)

We investigate the impact of high-order moments on the learning dynamics of an online Independent Component Analysis (ICA) algorithm under a high-dimensional data model composed of a weighted sum of two non-Gaussian random variables. This model allows precise control of the input moment structure via a weighting parameter. Building on an existing ordinary differential equation (ODE)-based analysis in the high-dimensional limit, we demonstrate that as the high-order moments increase, the algorithm exhibits slower convergence and demands both a lower learning rate and greater initial alignment to achieve informative solutions. Our findings highlight the algorithm's sensitivity to the statistical structure of the input data, particularly its moment characteristics. Furthermore, the ODE framework reveals a critical learning rate threshold necessary for learning when moments approach their maximum. These insights motivate future directions in moment-aware initialization and adaptive learning rate strategies to counteract the degradation in learning speed caused by high non-Gaussianity, thereby enhancing the robustness and efficiency of ICA in complex, high-dimensional settings.