# HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasets

arXiv: https://arxiv.org/abs/2304.03543 · HTML: https://mlsanity.com/paper/2304.03543/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Adversarial Generation of Continuous Images](https://mlsanity.com/paper/2011.12026/) (2020)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Hypernetwork functional image representation](https://mlsanity.com/paper/1902.10404/) (2019)
- [Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation](https://mlsanity.com/paper/2006.14284/) (2020)
- [Hypernetwork approach to generating point clouds](https://mlsanity.com/paper/2003.00802/) (2020)
- [Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain](https://mlsanity.com/paper/2108.12296/) (2021)
- [Hopular: Modern Hopfield Networks for Tabular Data](https://mlsanity.com/paper/2206.00664/) (2022)
- [PyHopper -- Hyperparameter optimization](https://mlsanity.com/paper/2210.04728/) (2022)

## Built on by (most cited first, 10 of 10)

- [HYTREL: Hypergraph-enhanced Tabular Data Representation Learning](https://mlsanity.com/paper/2307.08623/) (2023)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling](https://mlsanity.com/paper/2403.13319/) (2024)
- [Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks](https://mlsanity.com/paper/2310.00789/) (2023)
- [Interpretable Mesomorphic Networks for Tabular Data](https://mlsanity.com/paper/2305.13072/) (2023)
- [VisTabNet: Adapting Vision Transformers for Tabular Data](https://mlsanity.com/paper/2501.00057/) (2024)
- [Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nesting](https://mlsanity.com/paper/2502.15819/) (2025)
- [AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks](https://mlsanity.com/paper/2511.20170/) (2025)
- [Enhancing Dimensionality Prediction in Hybrid Metal Halides via Feature Engineering and Class-Imbalance Mitigation](https://mlsanity.com/paper/2512.05367/) (2025)
- [Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups](https://mlsanity.com/paper/2601.13094/) (2026)

## Abstract (arXiv)

Deep learning has achieved impressive performance in many domains, such as computer vision and natural language processing, but its advantage over classical shallow methods on tabular datasets remains questionable. It is especially challenging to surpass the performance of tree-like ensembles, such as XGBoost or Random Forests, on small-sized datasets (less than 1k samples). To tackle this challenge, we introduce HyperTab, a hypernetwork-based approach to solving small sample problems on tabular datasets. By combining the advantages of Random Forests and neural networks, HyperTab generates an ensemble of neural networks, where each target model is specialized to process a specific lower-dimensional view of the data. Since each view plays the role of data augmentation, we virtually increase the number of training samples while keeping the number of trainable parameters unchanged, which prevents model overfitting. We evaluated HyperTab on more than 40 tabular datasets of a varying number of samples and domains of origin, and compared its performance with shallow and deep learning models representing the current state-of-the-art. We show that HyperTab consistently outranks other methods on small data (with a statistically significant difference) and scores comparable to them on larger datasets. We make a python package with the code available to download at https://pypi.org/project/hypertab/