# Approximate Message Passing for Multi-Layer Estimation in Rotationally Invariant Models

arXiv: https://arxiv.org/abs/2212.01572 · HTML: https://mlsanity.com/paper/2212.01572/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 3 Dec 2022.

## Builds on (most cited first, 10 of 10)

- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Compressed Sensing using Generative Models](https://mlsanity.com/paper/1703.03208/) (2017)
- [Entropy and mutual information in models of deep neural networks](https://mlsanity.com/paper/1805.09785/) (2018)
- [A unifying tutorial on Approximate Message Passing](https://mlsanity.com/paper/2105.02180/) (2021)
- [Approximate Message Passing algorithms for rotationally invariant matrices](https://mlsanity.com/paper/2008.11892/) (2020)
- [Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing](https://mlsanity.com/paper/2112.04330/) (2021)
- [Inference with Deep Generative Priors in High Dimensions](https://mlsanity.com/paper/1911.03409/) (2019)
- [Multi-Layer Generalized Linear Estimation](https://mlsanity.com/paper/1701.06981/) (2017)
- [The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?](https://mlsanity.com/paper/2205.10009/) (2022)
- [A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines](https://mlsanity.com/paper/2005.01560/) (2020)

## Built on by (most cited first, 3 of 3)

- [ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data](https://mlsanity.com/paper/2406.11666/) (2024)
- [Unifying AMP Algorithms for Rotationally-Invariant Models](https://mlsanity.com/paper/2412.01574/) (2024)
- [Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing](https://mlsanity.com/paper/2604.27883/) (2026)

## Abstract (arXiv)

We consider the problem of reconstructing the signal and the hidden variables from observations coming from a multi-layer network with rotationally invariant weight matrices. The multi-layer structure models inference from deep generative priors, and the rotational invariance imposed on the weights generalizes the i.i.d. Gaussian assumption by allowing for a complex correlation structure, which is typical in applications. In this work, we present a new class of approximate message passing (AMP) algorithms and give a state evolution recursion which precisely characterizes their performance in the large system limit. In contrast with the existing multi-layer VAMP (ML-VAMP) approach, our proposed AMP -- dubbed multi-layer rotationally invariant generalized AMP (ML-RI-GAMP) -- provides a natural generalization beyond Gaussian designs, in the sense that it recovers the existing Gaussian AMP as a special case. Furthermore, ML-RI-GAMP exhibits a significantly lower complexity than ML-VAMP, as the computationally intensive singular value decomposition is replaced by an estimation of the moments of the design matrices. Finally, our numerical results show that this complexity gain comes at little to no cost in the performance of the algorithm.