Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models
Not on the map. Why?
Abstract (arXiv)
Identifying meaningful feature interactions is a central challenge in building accurate and interpretable models for tabular data. Generalized additive models (GAMs) have shown great success at modeling tabular data, but often rely on heuristic procedures to select interactions, potentially missing higher-order or context-dependent effects. To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods. Our key intuition is that tabular foundation models implicitly learn rich, adaptive feature dependencies through large-scale representation learning. Given a dataset, TabDistill first fits a tabular foundation model to the dataset, and then applies a post-hoc interaction attribution method to extract salient feature interactions from it. We evaluate these interactions by then using them as terms in a GAM. Across tasks, we find that interactions identified by TabDistill lead to consistent improvements in downstream GAMs' predictive performance. Our results suggest that tabular foundation models can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
38 of 58 references.
References · 38
- 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
- 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 318
- 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
- 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
- 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 243
- 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 152
- 2019The Shapley Taylor Interaction Indexcited by 143
- 2017PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparisoncited by 139
- 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 115
- 2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
- 2020GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactionscited by 82
- 2017Detecting Statistical Interactions from Neural Network Weightscited by 80
26 more
- 2022Faith-Shap: The Faithful Shapley Interaction Indexcited by 68
- 2017Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillationcited by 62
- 2022Neural Basis Models for Interpretabilitycited by 56
- 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 46
- 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
- 2017Iterative Random Forests to detect predictive and stable high-order interactionscited by 42
- 2018Improving the Interpretability of Deep Neural Networks with Knowledge Distillationcited by 37
- 2022Augmenting Interpretable Models with LLMs during Trainingcited by 36
- 2024A Closer Look at Deep Learning Methods on Tabular Datasetscited by 36
- 2021Adaptive wavelet distillation from neural networks through interpretationscited by 32
- 2025Foundation Models for Causal Inference via Prior-Data Fitted Networkscited by 30
- 2022Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selectioncited by 28
- 2022Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsitycited by 25
- 2022Scalable Interpretability via Polynomialscited by 25
- 2021GAM Changer: Editing Generalized Additive Models with Interactive Visualizationcited by 18
- 2025ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMscited by 16
- 2023Improving Neural Additive Models with Bayesian Principlescited by 15
- 2025Black Box Causal Inference: Effect Estimation via Meta Predictioncited by 13
- 2025SPEX: Scaling Feature Interaction Explanations for LLMscited by 12
- 2024InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretationcited by 10
- 2026Human-AI Co-design for Clinical Prediction Modelscited by 9
- 2023On the Impact of Knowledge Distillation for Model Interpretabilitycited by 8
- 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognitioncited by 7
- 2024Learning a Decision Tree Algorithm with Transformerscited by 6
- 2023Distilling BlackBox to Interpretable models for Efficient Transfer Learningcited by 4
- 2025"So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agentscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.