Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average

Normalising flows provide a powerful variational family for approximate inference, yet individual architectures often fail to generalise across heterogeneous posterior geometries. We revisit mixture-based flow formulations and introduce emphAMFmbox-VImbox-sEMA, a two-stage framework featuring a stable global weighting mechanism based on a Simplex Exponential Moving Average (sEMA) update. In Stage~1, a heterogeneous set of experts (textscRealNVP, textscMAF, textscRBIG) are trained independently to specialise in distinct structural regimes. In Stage~2, expert parameters are frozen and global mixture weights are learned through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update on the probability simplex. This design produces a tractable, data-agnostic gating mechanism (without per-sample gating or gradient backpropagation through weights) that adaptively reallocates capacity while avoiding component collapse. We evaluate the framework on ten posterior benchmarks: six canonical 2D synthetic families (Banana, X-Shaped, Bimodal, Multimodal, Two-moons, Rings) and four real/low-dimensional Bayesian targets (BLR, BPR, Weibull, Real-GMM2), with stronger baselines (textscNICE, textscResFlow, and EM-Mixing). Comprehensive evaluation covers NLL, KL divergence, Wasserstein-2 distance, and MMD, together with diagnostics of mixture dynamics, hyperparameter sensitivity, and cross-seed robustness. Empirically, emphAMFmbox-VImbox-sEMA achieves consistent NLL improvements over its predecessor emphAMFmbox-VI and avoids the catastrophic transport failures of single-flow baselines, while maintaining stable weight trajectories (Neff>1.4 on all datasets) with minimal computational overhead.