# Model Fusion with Kullback--Leibler Divergence

arXiv: https://arxiv.org/abs/2007.06168 · HTML: https://mlsanity.com/paper/2007.06168/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2210.06671/) (2022)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [Uncertainty Estimation in Multi-Agent Distributed Learning for AI-Enabled Edge Devices](https://mlsanity.com/paper/2403.09141/) (2024)
- [eFedDNN: Ensemble based Federated Deep Neural Networks for Trajectory Mode Inference](https://mlsanity.com/paper/2205.05756/) (2022)
- [Bayesian Federated Neural Matching that Completes Full Information](https://mlsanity.com/paper/2211.08010/) (2022)
- [ScoreFusion: Fusing Score-based Generative Models via Kullback-Leibler Barycenters](https://mlsanity.com/paper/2406.19619/) (2024)
- [Erasure Coded Neural Network Inference via Fisher Averaging](https://mlsanity.com/paper/2409.01420/) (2024)
- [Edge AI Collaborative Learning: Bayesian Approaches to Uncertainty Estimation](https://mlsanity.com/paper/2410.08651/) (2024)
- [Simplex Clustering via sBeta with Applications to Online Adjustment of Black-Box Predictions](https://mlsanity.com/paper/2208.00287/) (2022)
- [Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks](https://mlsanity.com/paper/2210.12974/) (2022)

## Abstract (arXiv)

We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and competitive with state-of-the-art on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.