# Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models

arXiv: https://arxiv.org/abs/2604.13332 · HTML: https://mlsanity.com/paper/2604.13332/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 30 of 38)

- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](https://mlsanity.com/paper/2004.13912/) (2020)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [The Shapley Taylor Interaction Index](https://mlsanity.com/paper/1902.05622/) (2019)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning](https://mlsanity.com/paper/2106.01613/) (2021)
- [GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactions](https://mlsanity.com/paper/2003.07132/) (2020)
- [Detecting Statistical Interactions from Neural Network Weights](https://mlsanity.com/paper/1705.04977/) (2017)
- [Faith-Shap: The Faithful Shapley Interaction Index](https://mlsanity.com/paper/2203.00870/) (2022)
- [Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation](https://mlsanity.com/paper/1710.06169/) (2017)
- [Neural Basis Models for Interpretability](https://mlsanity.com/paper/2205.14120/) (2022)
- [Do-PFN: In-Context Learning for Causal Effect Estimation](https://mlsanity.com/paper/2506.06039/) (2025)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Iterative Random Forests to detect predictive and stable high-order interactions](https://mlsanity.com/paper/1706.08457/) (2017)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [Improving the Interpretability of Deep Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/1812.10924/) (2018)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [Adaptive wavelet distillation from neural networks through interpretations](https://mlsanity.com/paper/2107.09145/) (2021)
- [Foundation Models for Causal Inference via Prior-Data Fitted Networks](https://mlsanity.com/paper/2506.10914/) (2025)
- [Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection](https://mlsanity.com/paper/2209.09326/) (2022)
- [Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity](https://mlsanity.com/paper/2202.12482/) (2022)
- [Scalable Interpretability via Polynomials](https://mlsanity.com/paper/2205.14108/) (2022)
- [GAM Changer: Editing Generalized Additive Models with Interactive Visualization](https://mlsanity.com/paper/2112.03245/) (2021)
- [ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs](https://mlsanity.com/paper/2505.17495/) (2025)
- [Improving Neural Additive Models with Bayesian Principles](https://mlsanity.com/paper/2305.16905/) (2023)
- [Black Box Causal Inference: Effect Estimation via Meta Prediction](https://mlsanity.com/paper/2503.05985/) (2025)

## Built on by (most cited first, 2 of 2)

- [GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models](https://mlsanity.com/paper/2608.18849/) (2026)
- [Coding LLMs Can Be Harnessed to Learn Interpretable Models from Data](https://mlsanity.com/paper/2610.14006/) (2026)

## Abstract (arXiv)

Identifying meaningful feature interactions is a central challenge in building accurate and interpretable models for tabular data. Generalized additive models (GAMs) have shown great success at modeling tabular data, but often rely on heuristic procedures to select interactions, potentially missing higher-order or context-dependent effects. To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods. Our key intuition is that tabular foundation models implicitly learn rich, adaptive feature dependencies through large-scale representation learning. Given a dataset, TabDistill first fits a tabular foundation model to the dataset, and then applies a post-hoc interaction attribution method to extract salient feature interactions from it. We evaluate these interactions by then using them as terms in a GAM. Across tasks, we find that interactions identified by TabDistill lead to consistent improvements in downstream GAMs' predictive performance. Our results suggest that tabular foundation models can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.