mlsanityData to 14 Oct 2026

Fine-Tuning the Retrieval Mechanism for Tabular Deep Learning

arXiv 2311.07343 ↗First posted 13 Nov 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

While interests in tabular deep learning has significantly grown, conventional tree-based models still outperform deep learning methods. To narrow this performance gap, we explore the innovative retrieval mechanism, a methodology that allows neural networks to refer to other data points while making predictions. Our experiments reveal that retrieval-based training, especially when fine-tuning the pretrained TabPFN model, notably surpasses existing methods. Moreover, the extensive pretraining plays a crucial role to enhance the performance of the model. These insights imply that blending the retrieval mechanism with pretraining and transfer learning schemes offers considerable potential for advancing the field of tabular deep learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

20 of 27 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  6. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  7. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  8. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  9. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  10. 2021Transformers Can Do Bayesian Inferencecited by 360
  11. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  12. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
8 more
  1. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  2. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  3. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  4. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 113
  5. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  6. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 49
  7. 2021Retrieval & Interaction Machine for Tabular Data Predictioncited by 21
  8. 2023Towards Cross-Table Masked Pretraining for Web Data Miningcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  2. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 20
  3. 2025On Finetuning Tabular Foundation Modelscited by 13
  4. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 8
  5. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 7
  6. 2026Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Modelscited by 3