# AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks

arXiv: https://arxiv.org/abs/2511.20170 · HTML: https://mlsanity.com/paper/2511.20170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [A Close Look at Deep Learning with Small Data](https://mlsanity.com/paper/2003.12843/) (2020)
- [HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasets](https://mlsanity.com/paper/2304.03543/) (2023)

## Abstract (arXiv)

Neural networks struggle on small tabular datasets, where tree-based models remain dominant. We introduce Adaptive Contrastive Approach (AdaCap), a training scheme that combines a permutation-based contrastive loss with a Tikhonov-based closed-form output mapping. Across 85 real-world regression datasets and multiple architectures, AdaCap yields consistent and statistically significant improvements in the small-sample regime, particularly for residual models. A meta-predictor trained on dataset characteristics (size, skewness, noise) accurately anticipates when AdaCap is beneficial. These results show that AdaCap acts as a targeted regularization mechanism, strengthening neural networks precisely where they are most fragile. All results and code are publicly available at https://github.com/BrunoBelucci/adacap.