# On approximating nabla f with neural networks

arXiv: https://arxiv.org/abs/1910.12744 · HTML: https://mlsanity.com/paper/1910.12744/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2019.

## Built on by (most cited first, 18 of 18)

- [Gradient Step Denoiser for convergent Plug-and-Play](https://mlsanity.com/paper/2110.03220/) (2021)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [The Monge Gap: A Regularizer to Learn All Transport Maps](https://mlsanity.com/paper/2302.04953/) (2023)
- [On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models](https://mlsanity.com/paper/2402.03845/) (2024)
- [Efficient Gradient Flows in Sliced-Wasserstein Space](https://mlsanity.com/paper/2110.10972/) (2021)
- [Input Convex Gradient Networks](https://mlsanity.com/paper/2111.12187/) (2021)
- [On Investigating the Conservative Property of Score-Based Generative Models](https://mlsanity.com/paper/2209.12753/) (2022)
- [Multimeasurement Generative Models](https://mlsanity.com/paper/2112.09822/) (2021)
- [Learning Gradients of Convex Functions with Monotone Gradient Networks](https://mlsanity.com/paper/2301.10862/) (2023)
- [Moment Matching Denoising Gibbs Sampling](https://mlsanity.com/paper/2305.11650/) (2023)
- [Convexity in ReLU Neural Networks: beyond ICNNs?](https://mlsanity.com/paper/2501.03017/) (2025)
- [Combining complex Langevin dynamics with score-based and energy-based diffusion models](https://mlsanity.com/paper/2510.01328/) (2025)
- [Compatible Gradient Approximations for Actor-Critic Algorithms](https://mlsanity.com/paper/2409.01477/) (2024)
- [Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors](https://mlsanity.com/paper/2606.06179/) (2026)
- [Leveraging Optimal Transport via Projections on Subspaces for Machine Learning Applications](https://mlsanity.com/paper/2311.13883/) (2023)
- [Classification-Denoising Networks](https://mlsanity.com/paper/2410.03505/) (2024)
- [Gradient Networks](https://mlsanity.com/paper/2404.07361/) (2024)
- [Universal Representation of Generalized Convex Functions and their Gradients](https://mlsanity.com/paper/2509.04477/) (2025)

## Abstract (arXiv)

Consider a feedforward neural network psi: ℝdrightarrow ℝd such that psi≈ nabla f, where f:ℝd rightarrow ℝ is a smooth function, therefore psi must satisfy partialj psii = partiali psij pointwise. We prove a theorem that a psi network with more than one hidden layer can only represent one feature in its first hidden layer; this is a dramatic departure from the well-known results for one hidden layer. The proof of the theorem is straightforward, where two backward paths and a weight-tying matrix play the key roles. We then present the alternative, the implicit parametrization, where the neural network is φ: ℝd rightarrow ℝ and nabla φ ≈ nabla f; in addition, a "soft analysis" of nabla φ gives a dual perspective on the theorem. Throughout, we come back to recent probabilistic models that are formulated as nabla φ ≈ nabla f, and conclude with a critique of denoising autoencoders.