# Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks

arXiv: https://arxiv.org/abs/1802.02195 · HTML: https://mlsanity.com/paper/1802.02195/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 31 of its 73 references that are in the map.
- First posted on arXiv 6 Feb 2018.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 30 of 31)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Multi-task Self-Supervised Visual Learning](https://mlsanity.com/paper/1708.07860/) (2017)
- [Conditional Computation in Neural Networks for faster models](https://mlsanity.com/paper/1511.06297/) (2015)
- [Reasoning about Entailment with Neural Attention](https://mlsanity.com/paper/1509.06664/) (2015)
- [Beyond Sparsity: Tree Regularization of Deep Models for Interpretability](https://mlsanity.com/paper/1711.06178/) (2017)
- [MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network](https://mlsanity.com/paper/1707.02485/) (2017)
- [Auditing Black-box Models for Indirect Influence](https://mlsanity.com/paper/1602.07043/) (2016)
- [Deep Networks with Internal Selective Attention through Feedback Connections](https://mlsanity.com/paper/1407.3068/) (2014)
- [Classification of Radiology Reports Using Neural Attention Models](https://mlsanity.com/paper/1708.06828/) (2017)

## Built on by (most cited first, 13 of 13)

- [Causal Discovery from Temporal Data: An Overview and New Perspectives](https://mlsanity.com/paper/2303.10112/) (2023)
- [Neural Markov Jump Processes](https://mlsanity.com/paper/2305.19744/) (2023)
- [Inductive Granger Causal Modeling for Multivariate Time Series](https://mlsanity.com/paper/2102.05298/) (2021)
- [Instance-wise Causal Feature Selection for Model Interpretation](https://mlsanity.com/paper/2104.12759/) (2021)
- [Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning](https://mlsanity.com/paper/2112.03020/) (2021)
- [A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts](https://mlsanity.com/paper/1907.05346/) (2019)
- [Learning Channel Importance for High Content Imaging with Interpretable Deep Input Channel Mixing](https://mlsanity.com/paper/2308.16637/) (2023)
- [Causal Feature Selection for Responsible Machine Learning](https://mlsanity.com/paper/2402.02696/) (2024)
- [Training Deep Models to be Explained with Fewer Examples](https://mlsanity.com/paper/2112.03508/) (2021)
- [Linking Model Intervention to Causal Interpretation in Model Explanation](https://mlsanity.com/paper/2410.15648/) (2024)
- [TPK: Trustworthy Trajectory Prediction Integrating Prior Knowledge For Interpretability and Kinematic Feasibility](https://mlsanity.com/paper/2505.06743/) (2025)
- [Towards a Universal Gating Network for Mixtures of Experts](https://mlsanity.com/paper/2011.01613/) (2020)
- [Explanation-based Training with Differentiable Insertion/Deletion Metric-aware Regularizers](https://mlsanity.com/paper/2310.12553/) (2023)

## Abstract (arXiv)

Knowledge of the importance of input features towards decisions made by machine-learning models is essential to increase our understanding of both the models and the underlying data. Here, we present a new approach to estimating feature importance with neural networks based on the idea of distributing the features of interest among experts in an attentive mixture of experts (AME). AMEs use attentive gating networks trained with a Granger-causal objective to learn to jointly produce accurate predictions as well as estimates of feature importance in a single model. Our experiments show (i) that the feature importance estimates provided by AMEs compare favourably to those provided by state-of-the-art methods, (ii) that AMEs are significantly faster at estimating feature importance than existing methods, and (iii) that the associations discovered by AMEs are consistent with those reported by domain experts.