# Mitigating Representation Gaps in Amortized Bayesian Inference with Auxiliary Supervision

arXiv: https://arxiv.org/abs/2609.39525 · HTML: https://mlsanity.com/paper/2609.39525/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 27 of 27)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [BayesFlow: Learning complex stochastic models with invertible neural networks](https://mlsanity.com/paper/2003.06281/) (2020)
- [Validating Bayesian Inference Algorithms with Simulation-Based Calibration](https://mlsanity.com/paper/1804.06788/) (2018)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [Neural Approximate Sufficient Statistics for Implicit Models](https://mlsanity.com/paper/2010.10079/) (2020)
- [Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks](https://mlsanity.com/paper/2112.08866/) (2021)
- [Learning Robust Statistics for Simulation-based Inference under Model Misspecification](https://mlsanity.com/paper/2305.15871/) (2023)
- [Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network](https://mlsanity.com/paper/1510.02175/) (2015)
- [Likelihood-Free Parameter Estimation with Neural Bayes Estimators](https://mlsanity.com/paper/2208.12942/) (2022)
- [Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation](https://mlsanity.com/paper/2208.13624/) (2022)
- [Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity](https://mlsanity.com/paper/2211.02383/) (2022)
- [The Statistical Accuracy of Neural Posterior and Likelihood Estimation](https://mlsanity.com/paper/2411.12068/) (2024)
- [Toward a principled Bayesian workflow in cognitive science](https://mlsanity.com/paper/1904.12765/) (2019)
- [Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning](https://mlsanity.com/paper/1909.02005/) (2019)
- [Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability](https://mlsanity.com/paper/2310.13402/) (2023)
- [Robust Amortized Bayesian Inference with Self-Consistency Losses on Unlabeled Data](https://mlsanity.com/paper/2501.13483/) (2025)
- [Diffusion Models in Simulation-Based Inference: A Tutorial Review](https://mlsanity.com/paper/2512.20685/) (2025)
- [Balancing Simulation-based Inference for Conservative Posteriors](https://mlsanity.com/paper/2304.10978/) (2023)
- [Does Unsupervised Domain Adaptation Improve the Robustness of Amortized Bayesian Inference? A Systematic Evaluation](https://mlsanity.com/paper/2502.04949/) (2025)
- [Domain-Adaptive Neural Posterior Estimation for Strong Gravitational Lens Analysis](https://mlsanity.com/paper/2410.16347/) (2024)
- [Minimum Distance Summaries for Robust Neural Posterior Estimation](https://mlsanity.com/paper/2602.09161/) (2026)
- [Theoretical guarantees for neural estimators in parametric statistics](https://mlsanity.com/paper/2506.18508/) (2025)
- [From Mice to Trains: Amortized Bayesian Inference on Graph Data](https://mlsanity.com/paper/2601.02241/) (2026)

## Abstract (arXiv)

Casting Bayesian inference as a neural network optimization problem targeting an amortized posterior is attractive, as it extends to otherwise intractable statistical models and offers near instantaneous inference for new datasets after prepaying the training cost. Although theory guarantees faithfulness under ideal convergence, practical amortized inference still requires iterating over architectures and optimization choices and ultimately ``satisficing'' under finite simulation, compute, and time budgets. Even the best-performing solution may thus retain avoidable representation gaps that typically require problem-specific fixes. Here, we propose a generic alternative which improves training dynamics with auxiliary guidance losses applied to internal representations. Specifically, we show how such guidance leads to faster convergence when training data is abundant and to better performance when it is scarce. We formalize representation gaps as getting stuck in a local optimum at the information bottleneck between the parts of the network tasked with feature learning and those tasked with conditional distribution learning, and offer a generic diagnostic to separate summary failures from inference failures. Finally, we demonstrate that auxiliary supervision improves convergence speed and accuracy on a range of challenging real-world inference problems.