# Functional Gradient Descent with Adaptive Representations

arXiv: https://arxiv.org/abs/2606.16926 · HTML: https://mlsanity.com/paper/2606.16926/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization](https://mlsanity.com/paper/1905.01332/) (2019)
- [Functional Bilevel Optimization for Machine Learning](https://mlsanity.com/paper/2403.20233/) (2024)
- [Nonparametric Instrumental Variable Regression through Stochastic Approximate Gradients](https://mlsanity.com/paper/2402.05639/) (2024)
- [Statistical Learning and Inverse Problems: A Stochastic Gradient Approach](https://mlsanity.com/paper/2209.14967/) (2022)
- [Neuro-Spectral Architectures for Causal Physics-Informed Networks](https://mlsanity.com/paper/2509.04966/) (2025)
- [Sobolev Gradient Ascent for Optimal Transport: Barycenter Optimization and Convergence Analysis](https://mlsanity.com/paper/2505.13660/) (2025)
- [Random Gradient-Free Optimization in Infinite Dimensional Spaces](https://mlsanity.com/paper/2512.20566/) (2025)

## Built on by (most cited first, 1 of 1)

- [CAGE-NAS: Certified Functional Descent for Efficient Model Growth](https://mlsanity.com/paper/2610.01173/) (2026)

## Abstract (arXiv)

Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis. An interesting alternative is functional gradient descent (FGD), that is, gradient descent directly in function space, which benefits from strong convergence results and admits a clean theory. However, FGD is difficult to implement in practice because functional gradients are infinite-dimensional, and thus cannot be fully computed nor stored in memory. Existing implementations therefore rely on fixed approximations, which introduce approximation error. We propose a new, theoretically-grounded FGD algorithm that adapts the representation of the functional gradients over the course of optimization. By explicitly incorporating this approximation into the analysis, we establish convergence to a stationary point (for smooth losses) and to a global minimizer (under smoothness + a Polyak-Lojasiewicz-type condition) regardless of our approximations. To the best of our knowledge, this is the first implementable FGD method with such guarantees in a general setting. We demonstrate the effectiveness of our method on regression, numerical solution of PDEs, and modern computer vision. Across settings, our method consistently outperforms both FGD with fixed approximations and neural network baselines in efficiency and accuracy.