# Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation

arXiv: https://arxiv.org/abs/2006.14284 · HTML: https://mlsanity.com/paper/2006.14284/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 27 of 27)

- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Knowledge Distillation as Semiparametric Inference](https://mlsanity.com/paper/2104.09732/) (2021)
- [Benchmarking Multimodal AutoML for Tabular Data with Text Fields](https://mlsanity.com/paper/2111.02705/) (2021)
- [Gradient-Free Structured Pruning with Unlabeled Data](https://mlsanity.com/paper/2303.04185/) (2023)
- [Flexible Model Aggregation for Quantile Regression](https://mlsanity.com/paper/2103.00083/) (2021)
- [Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey](https://mlsanity.com/paper/2505.01821/) (2025)
- [Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement](https://mlsanity.com/paper/2303.08983/) (2023)
- [Synthesising Multi-Modal Minority Samples for Tabular Data](https://mlsanity.com/paper/2105.08204/) (2021)
- [Time-series Transformer Generative Adversarial Networks](https://mlsanity.com/paper/2205.11164/) (2022)
- [HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasets](https://mlsanity.com/paper/2304.03543/) (2023)
- [Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features](https://mlsanity.com/paper/2110.13413/) (2021)
- [Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing](https://mlsanity.com/paper/2109.11105/) (2021)
- [In Search of netUnicorn: A Data-Collection Platform to Develop Generalizable ML Models for Network Security Problems](https://mlsanity.com/paper/2306.08853/) (2023)
- [Strong statistical parity through fair synthetic data](https://mlsanity.com/paper/2311.03000/) (2023)
- [Demonstrating Large-Scale Package Manipulation via Learned Metrics of Pick Success](https://mlsanity.com/paper/2305.10272/) (2023)
- [Learning Capacity: A Measure of the Effective Dimensionality of a Model](https://mlsanity.com/paper/2305.17332/) (2023)
- [Bringing a Ruler Into the Black Box: Uncovering Feature Impact from Individual Conditional Expectation Plots](https://mlsanity.com/paper/2109.02724/) (2021)
- [A Robust Stacking Framework for Training Deep Graph Models with Multifaceted Node Features](https://mlsanity.com/paper/2206.08473/) (2022)
- [Tiny Classifier Circuits: Evolving Accelerators for Tabular Data](https://mlsanity.com/paper/2303.00031/) (2023)
- [Scheduled Knowledge Acquisition on Lightweight Vector Symbolic Architectures for Brain-Computer Interfaces](https://mlsanity.com/paper/2403.13844/) (2024)
- [Pedagogical Rule Extraction to Learn Interpretable Models - an Empirical Study](https://mlsanity.com/paper/2112.13285/) (2021)
- [Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method](https://mlsanity.com/paper/2508.14783/) (2025)
- [Boosting Predictive Performance on Tabular Data through Data Augmentation with Latent-Space Flow-Based Diffusion](https://mlsanity.com/paper/2511.16571/) (2025)
- [SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling](https://mlsanity.com/paper/2609.12455/) (2026)
- [Distillation of Tabular Foundation Models into Efficient Predictors](https://mlsanity.com/paper/2610.01435/) (2026)

## Abstract (arXiv)

Automated machine learning (AutoML) can produce complex model ensembles by stacking, bagging, and boosting many individual models like trees, deep networks, and nearest neighbor estimators. While highly accurate, the resulting predictors are large, slow, and opaque as compared to their constituents. To improve the deployment of AutoML on tabular data, we propose FAST-DAD to distill arbitrarily complex ensemble predictors into individual models like boosted trees, random forests, and deep networks. At the heart of our approach is a data augmentation strategy based on Gibbs sampling from a self-attention pseudolikelihood estimator. Across 30 datasets spanning regression and binary/multiclass classification tasks, FAST-DAD distillation produces significantly better individual models than one obtains through standard training on the original data. Our individual distilled models are over 10x faster and more accurate than ensemble predictors produced by AutoML tools like H2O/AutoSklearn.