# Variational Gaussian Approximation in Replica Analysis of Parametric Models

arXiv: https://arxiv.org/abs/2509.11780 · HTML: https://mlsanity.com/paper/2509.11780/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 20 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 4 of 4)

- [A General Framework for Updating Belief Distributions](https://mlsanity.com/paper/1306.6430/) (2013)
- [A modern maximum-likelihood theory for high-dimensional logistic regression](https://mlsanity.com/paper/1803.06964/) (2018)
- [A statistical mechanics approach to de-biasing and uncertainty estimation in LASSO for random measurements](https://mlsanity.com/paper/1803.09927/) (2018)
- [Posterior Covariance Information Criterion for Weighted Inference](https://mlsanity.com/paper/2106.13694/) (2021)

## Abstract (arXiv)

We revisit the replica method for analyzing inference and learning in parametric models, considering situations where the data-generating distribution is unknown or analytically intractable. Instead of assuming idealized distributions to carry out quenched averages analytically, we use a variational Gaussian approximation for the replicated system in grand canonical formalism in which the data average can be deferred and replaced by empirical averages, leading to stationarity conditions that adaptively determine the parameters of the trial Hamiltonian for each dataset. This approach clarifies how fluctuations affect information extraction and connects directly with the results of mathematical statistics or learning theory such as information criteria. As a concrete application, we analyze linear regression and derive learning curves. This includes cases with real-world datasets, where exact replica calculations are not feasible.