# Surfing: Iterative optimization over incrementally trained deep networks

arXiv: https://arxiv.org/abs/1907.08653 · HTML: https://mlsanity.com/paper/1907.08653/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2019.

## Built on by (most cited first, 5 of 5)

- [Intermediate Layer Optimization for Inverse Problems using Deep Generative Models](https://mlsanity.com/paper/2102.07364/) (2021)
- [Instance-Optimal Compressed Sensing via Posterior Sampling](https://mlsanity.com/paper/2106.11438/) (2021)
- [Optimization for Amortized Inverse Problems](https://mlsanity.com/paper/2210.13983/) (2022)
- [Signal Recovery with Non-Expansive Generative Network Priors](https://mlsanity.com/paper/2204.13599/) (2022)
- [A Linearly Convergent GAN Inversion-based Algorithm for Reverse Engineering of Deceptions](https://mlsanity.com/paper/2306.04756/) (2023)

## Abstract (arXiv)

We investigate a sequential optimization procedure to minimize the empirical risk functional fhatθ(x) = frac12|Ghatθ(x) - y|2 for certain families of deep networks Gθ(x). The approach is to optimize a sequence of objective functions that use network parameters obtained during different stages of the training process. When initialized with random parameters θ0, we show that the objective fθ0(x) is "nice'' and easy to optimize with gradient descent. As learning is carried out, we obtain a sequence of generative networks x mapsto Gθt(x) and associated risk functions fθt(x), where t indicates a stage of stochastic gradient descent during training. Since the parameters of the network do not change by very much in each step, the surface evolves slowly and can be incrementally optimized. The algorithm is formalized and analyzed for a family of expansive networks. We call the procedure it surfing since it rides along the peak of the evolving (negative) empirical risk function, starting from a smooth surface at the beginning of learning and ending with a wavy nonconvex surface after learning is complete. Experiments show how surfing can be used to find the global optimum and for compressed sensing even when direct gradient descent on the final learned network fails.