# A Complete Decomposition of KL Error using Refined Information and Mode Interaction Selection

arXiv: https://arxiv.org/abs/2410.11964 · HTML: https://mlsanity.com/paper/2410.11964/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection](https://mlsanity.com/paper/2209.09326/) (2022)
- [Stratified Graphical Models - Context-Specific Independence in Graphical Models](https://mlsanity.com/paper/1309.6415/) (2013)
- [Legendre Decomposition for Tensors](https://mlsanity.com/paper/1802.04502/) (2018)
- [Low-Rank Approximation and Completion of Positive Tensors](https://mlsanity.com/paper/1412.0620/) (2014)
- [Many-body Approximation for Non-negative Tensors](https://mlsanity.com/paper/2209.15338/) (2022)
- [Feature Selection with Redundancy-complementariness Dispersion](https://mlsanity.com/paper/1502.00231/) (2015)
- [Interaction Pursuit with Feature Screening and Selection](https://mlsanity.com/paper/1605.08933/) (2016)
- [Towards Hybrid-grained Feature Interaction Selection for Deep Sparse Network](https://mlsanity.com/paper/2310.15342/) (2023)
- [Sparse Nested Markov models with Log-linear Parameters](https://mlsanity.com/paper/1309.6863/) (2013)

## Built on by (most cited first, 2 of 2)

- [Higher-Order Causal Structure Learning with Additive Models](https://mlsanity.com/paper/2511.03831/) (2025)
- [Differentiable Structure Learning and Causal Discovery for General Binary Data](https://mlsanity.com/paper/2509.21658/) (2025)

## Abstract (arXiv)

The log-linear model has received a significant amount of theoretical attention in previous decades and remains the fundamental tool used for learning probability distributions over discrete variables. Despite its large popularity in statistical mechanics and high-dimensional statistics, the majority of related energy-based models only focus on the two-variable relationships, such as Boltzmann machines and Markov graphical models. Although these approaches have easier-to-solve structure learning problems and easier-to-optimize parametric distributions, they often ignore the rich structure which exists in the higher-order interactions between different variables. Using more recent tools from the field of information geometry, we revisit the classical formulation of the log-linear model with a focus on higher-order mode interactions, going beyond the 1-body modes of independent distributions and the 2-body modes of Boltzmann distributions. This perspective allows us to define a complete decomposition of the KL error. This then motivates the formulation of a sparse selection problem over the set of possible mode interactions. In the same way as sparse graph selection allows for better generalization, we find that our learned distributions are able to more efficiently use the finite amount of data which is available in practice. We develop an algorithm called MAHGenTa which leverages a novel Monte-Carlo sampling technique for energy-based models alongside a greedy heuristic for incorporating statistical robustness. On both synthetic and real-world datasets, we demonstrate our algorithm's effectiveness in maximizing the log-likelihood for the generative task and also the ease of adaptability to the discriminative task of classification.