# Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks

arXiv: https://arxiv.org/abs/1907.00485 · HTML: https://mlsanity.com/paper/1907.00485/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2019.

## Built on by (most cited first, 10 of 10)

- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [An Exact Poly-Time Membership-Queries Algorithm for Extraction a three-Layer ReLU Network](https://mlsanity.com/paper/2105.09673/) (2021)
- [Going Beyond Linear RL: Sample Efficient Neural Function Approximation](https://mlsanity.com/paper/2107.06466/) (2021)
- [Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samples](https://mlsanity.com/paper/2101.07150/) (2021)
- [Landscape analysis of an improved power method for tensor decomposition](https://mlsanity.com/paper/2110.15821/) (2021)
- [Finite Sample Identification of Wide Shallow Neural Networks with Biases](https://mlsanity.com/paper/2211.04589/) (2022)
- [Efficient Identification of Butterfly Sparse Matrix Factorizations](https://mlsanity.com/paper/2110.01230/) (2021)
- [Low-Rank Tensor Decompositions for the Theory of Neural Networks](https://mlsanity.com/paper/2508.18408/) (2025)
- [Approximate Real Symmetric Tensor Rank](https://mlsanity.com/paper/2207.12529/) (2022)
- [Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks](https://mlsanity.com/paper/2409.19138/) (2024)

## Abstract (arXiv)

We address the structure identification and the uniform approximation of two fully nonlinear layer neural networks of the type f(x)=1T h(BT g(AT x)) on mathbb Rd from a small number of query samples. We approach the problem by sampling actively finite difference approximations to Hessians of the network. Gathering several approximate Hessians allows reliably to approximate the matrix subspace mathcal W spanned by symmetric tensors a1 otimes a1 ,dots,am0otimes am0 formed by weights of the first layer together with the entangled symmetric tensors v1 otimes v1 ,dots,vm1otimes vm1, formed by suitable combinations of the weights of the first and second layer as vℓ=A G0 bℓ/|A G0 bℓ|2, ℓ ∈ [m1], for a diagonal matrix G0 depending on the activation functions of the first layer. The identification of the 1-rank symmetric tensors within mathcal W is then performed by the solution of a robust nonlinear program. We provide guarantees of stable recovery under a posteriori verifiable conditions. We further address the correct attribution of approximate weights to the first or second layer. By using a suitably adapted gradient descent iteration, it is possible then to estimate, up to intrinsic symmetries, the shifts of the activations functions of the first layer and compute exactly the matrix G0. Our method of identification of the weights of the network is fully constructive, with quantifiable sample complexity, and therefore contributes to dwindle the black-box nature of the network training phase. We corroborate our theoretical results by extensive numerical experiments.