# Representing smooth functions as compositions of near-identity functions with implications for deep network optimization

arXiv: https://arxiv.org/abs/1804.05012 · HTML: https://mlsanity.com/paper/1804.05012/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2018.

## Built on by (most cited first, 15 of 15)

- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Architecture Matters in Continual Learning](https://mlsanity.com/paper/2202.00275/) (2022)
- [Overparameterization of deep ResNet: zero loss and mean-field analysis](https://mlsanity.com/paper/2105.14417/) (2021)
- [An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models](https://mlsanity.com/paper/2212.14852/) (2022)
- [Out of Distribution Data Detection Using Dropout Bayesian Neural Networks](https://mlsanity.com/paper/2202.08985/) (2022)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [On Measuring Excess Capacity in Neural Networks](https://mlsanity.com/paper/2202.08070/) (2022)
- [DeDUCE: Generating Counterfactual Explanations Efficiently](https://mlsanity.com/paper/2111.15639/) (2021)
- [Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks](https://mlsanity.com/paper/2204.07261/) (2022)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [An Entropy-Based Model for Hierarchical Learning](https://mlsanity.com/paper/2212.14681/) (2022)
- [Multi-Label Out-of-Distribution Detection with Spectral Normalized Joint Energy](https://mlsanity.com/paper/2405.04759/) (2024)
- [Smooth Model Compression without Fine-Tuning](https://mlsanity.com/paper/2505.24469/) (2025)
- [Transformed Latent Variable Multi-Output Gaussian Processes](https://mlsanity.com/paper/2605.05133/) (2026)
- [Prior-aware and Context-guided Group Sampling for Active Probabilistic Subsampling](https://mlsanity.com/paper/2607.07083/) (2026)

## Abstract (arXiv)

We show that any smooth bi-Lipschitz h can be represented exactly as a composition hm circ ... circ h1 of functions h1,...,hm that are close to the identity in the sense that each left(hi-Idright) is Lipschitz, and the Lipschitz constant decreases inversely with the number m of functions composed. This implies that h can be represented to any accuracy by a deep residual network whose nonlinear layers compute functions with a small Lipschitz constant. Next, we consider nonlinear regression with a composition of near-identity nonlinear maps. We show that, regarding Fr'echet derivatives with respect to the h1,...,hm, any critical point of a quadratic criterion in this near-identity region must be a global minimizer. In contrast, if we consider derivatives with respect to parameters of a fixed-size residual network with sigmoid activation functions, we show that there are near-identity critical points that are suboptimal, even in the realizable case. Informally, this means that functional gradient methods for residual networks cannot get stuck at suboptimal critical points corresponding to near-identity layers, whereas parametric gradient methods for sigmoidal residual networks suffer from suboptimal critical points in the near-identity region.