# Staying on the Manifold: Geometry-Aware Noise Injection

arXiv: https://arxiv.org/abs/2509.20201 · HTML: https://mlsanity.com/paper/2509.20201/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [An introduction to Topological Data Analysis: fundamental and practical aspects for data scientists](https://mlsanity.com/paper/1710.04019/) (2017)
- [Anticorrelated Noise Injection for Improved Generalization](https://mlsanity.com/paper/2202.02831/) (2022)
- [Explicit Regularization in Overparametrized Models via Noise Injection](https://mlsanity.com/paper/2206.04613/) (2022)
- [Neglected Hessian component explains mysteries in Sharpness regularization](https://mlsanity.com/paper/2401.10809/) (2024)
- [First-Order Manifold Data Augmentation for Regression Learning](https://mlsanity.com/paper/2406.10914/) (2024)
- [Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks](https://mlsanity.com/paper/2306.17630/) (2023)
- [Riemannian Generative Decoder](https://mlsanity.com/paper/2506.19133/) (2025)
- [Geometric Gaussian Approximations of Probability Distributions](https://mlsanity.com/paper/2507.00616/) (2025)

## Built on by (most cited first, 4 of 4)

- [Reducing Memorisation in Generative Models via Riemannian Bayesian Inference](https://mlsanity.com/paper/2602.00199/) (2026)
- [Enhancing classification accuracy through chaos](https://mlsanity.com/paper/2603.15299/) (2026)
- [Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective](https://mlsanity.com/paper/2605.21692/) (2026)
- [MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Prediction](https://mlsanity.com/paper/2606.30370/) (2026)

## Abstract (arXiv)

It has been shown that perturbing the input during training implicitly regularises the gradient of the learnt function, leading to smoother models and enhancing generalisation. However, previous research mostly considered the addition of ambient noise in the input space, without considering the underlying structure of the data. In this work, we propose several strategies of adding geometry-aware input noise that accounts for the lower dimensional manifold the input space inhabits. We start by projecting ambient Gaussian noise onto the tangent space of the manifold. In a second step, the noise sample is mapped on the manifold via the associated geodesic curve. We also consider Brownian motion noise, which moves in random steps along the manifold. We show that geometry-aware noise leads to improved generalisation and robustness to hyperparameter selection on highly curved manifolds, while performing at least as well as training without noise on simpler manifolds. Our proposed framework extends to data manifolds approximated by generative models and we observe similar trends on the MNIST digits dataset.