# TabKD: Tabular Knowledge Distillation through Interaction Diversity of Learned Feature Bins

arXiv: https://arxiv.org/abs/2603.15481 · HTML: https://mlsanity.com/paper/2603.15481/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Model extraction from counterfactual explanations](https://mlsanity.com/paper/2009.01884/) (2020)
- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [DualCF: Efficient Model Extraction Attack from Counterfactual Explanations](https://mlsanity.com/paper/2205.06504/) (2022)
- [TVAE: Triplet-Based Variational Autoencoder using Metric Learning](https://mlsanity.com/paper/1802.04403/) (2018)
- [Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack using Public Data](https://mlsanity.com/paper/2302.08466/) (2023)
- [Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation](https://mlsanity.com/paper/2402.12406/) (2024)
- [On Learning Representations for Tabular Data Distillation](https://mlsanity.com/paper/2501.13905/) (2025)

## Built on by (most cited first, 1 of 1)

- [Probabilistic Calibration Is a Trainable Capability in Language Models](https://mlsanity.com/paper/2605.11845/) (2026)

## Abstract (arXiv)

Data-free knowledge distillation enables model compression without original training data, critical for privacy-sensitive tabular domains. However, existing methods does not perform well on tabular data because they do not explicitly address feature interactions, the fundamental way tabular models encode predictive knowledge. We identify interaction diversity, systematic coverage of feature combinations, as an essential requirement for effective tabular distillation. To operationalize this insight, we propose TabKD, which learns adaptive feature bins aligned with teacher decision boundaries, then generates synthetic queries that maximize pairwise interaction coverage. Across 4 benchmark datasets and 4 teacher architectures, TabKD achieves highest student-teacher agreement in 14 out of 16 configurations, outperforming 5 state-of-the-art baselines. We further show that interaction coverage strongly correlates with distillation quality, validating our core hypothesis. Our work establishes interaction-focused exploration as a principled framework for tabular model extraction.