# Mixture of Linear Models Co-supervised by Deep Neural Networks

arXiv: https://arxiv.org/abs/2108.04035 · HTML: https://mlsanity.com/paper/2108.04035/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 55 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](https://mlsanity.com/paper/2004.13912/) (2020)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Interpretable Convolutional Neural Networks](https://mlsanity.com/paper/1710.00935/) (2017)
- [Detecting Statistical Interactions from Neural Network Weights](https://mlsanity.com/paper/1705.04977/) (2017)
- [Exploring Interpretable LSTM Neural Networks over Multi-Variable Data](https://mlsanity.com/paper/1905.12034/) (2019)
- [Explaining Deep Learning Models - A Bayesian Non-parametric Approach](https://mlsanity.com/paper/1811.03422/) (2018)
- [The return of AdaBoost.MH: multi-class Hamming trees](https://mlsanity.com/paper/1312.6086/) (2013)
- [An interpretable neural network model through piecewise linear approximation](https://mlsanity.com/paper/2001.07119/) (2020)

## Built on by (most cited first, 5 of 5)

- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [SynthTree: Co-supervised Local Model Synthesis for Explainable Prediction](https://mlsanity.com/paper/2406.10962/) (2024)
- [Cluster-Based Generalized Additive Models Informed by Random Fourier Features](https://mlsanity.com/paper/2512.19373/) (2025)
- [REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models](https://mlsanity.com/paper/2609.32141/) (2026)
- [Co-Supervised Tree Synthesis for Interpretable Subgroup Identification and Honest Treatment Effect Inference in Randomized Trials](https://mlsanity.com/paper/2610.03927/) (2026)

## Abstract (arXiv)

Deep neural network (DNN) models have achieved phenomenal success for applications in many domains, ranging from academic research in science and engineering to industry and business. The modeling power of DNN is believed to have come from the complexity and over-parameterization of the model, which on the other hand has been criticized for the lack of interpretation. Although certainly not true for every application, in some applications, especially in economics, social science, healthcare industry, and administrative decision making, scientists or practitioners are resistant to use predictions made by a black-box system for multiple reasons. One reason is that a major purpose of a study can be to make discoveries based upon the prediction function, e.g., to reveal the relationships between measurements. Another reason can be that the training dataset is not large enough to make researchers feel completely sure about a purely data-driven result. Being able to examine and interpret the prediction function will enable researchers to connect the result with existing knowledge or gain insights about new directions to explore. Although classic statistical models are much more explainable, their accuracy often falls considerably below DNN. In this paper, we propose an approach to fill the gap between relatively simple explainable models and DNN such that we can more flexibly tune the trade-off between interpretability and accuracy. Our main idea is a mixture of discriminative models that is trained with the guidance from a DNN. Although mixtures of discriminative models have been studied before, our way of generating the mixture is quite different.