# Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average

arXiv: https://arxiv.org/abs/2607.03809 · HTML: https://mlsanity.com/paper/2607.03809/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 4 Jul 2026.

## Builds on (most cited first, 8 of 8)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Iterative Gaussianization: from ICA to Random Rotations](https://mlsanity.com/paper/1602.00229/) (2016)
- [Variational Mixture of Normalizing Flows](https://mlsanity.com/paper/2009.00585/) (2020)

## Abstract (arXiv)

Normalising flows provide a powerful variational family for approximate inference, yet individual architectures often fail to generalise across heterogeneous posterior geometries. We revisit mixture-based flow formulations and introduce emphAMFmbox-VImbox-sEMA, a two-stage framework featuring a stable global weighting mechanism based on a Simplex Exponential Moving Average (sEMA) update. In Stage~1, a heterogeneous set of experts (textscRealNVP, textscMAF, textscRBIG) are trained independently to specialise in distinct structural regimes. In Stage~2, expert parameters are frozen and global mixture weights are learned through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update on the probability simplex. This design produces a tractable, data-agnostic gating mechanism (without per-sample gating or gradient backpropagation through weights) that adaptively reallocates capacity while avoiding component collapse. We evaluate the framework on ten posterior benchmarks: six canonical 2D synthetic families (Banana, X-Shaped, Bimodal, Multimodal, Two-moons, Rings) and four real/low-dimensional Bayesian targets (BLR, BPR, Weibull, Real-GMM2), with stronger baselines (textscNICE, textscResFlow, and EM-Mixing). Comprehensive evaluation covers NLL, KL divergence, Wasserstein-2 distance, and MMD, together with diagnostics of mixture dynamics, hyperparameter sensitivity, and cross-seed robustness. Empirically, emphAMFmbox-VImbox-sEMA achieves consistent NLL improvements over its predecessor emphAMFmbox-VI and avoids the catastrophic transport failures of single-flow baselines, while maintaining stable weight trajectories (Neff>1.4 on all datasets) with minimal computational overhead.