mlsanityData to 7 Oct 2026

Retrieval & Fine-Tuning for In-Context Tabular Models

arXiv 2406.05207 ↗First posted 7 Jun 2024cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Tabular data is a pervasive modality spanning a wide range of domains, and the inherent diversity poses a considerable challenge for deep learning. Recent advancements using transformer-based in-context learning have shown promise on smaller and less complex datasets, but have struggled to scale to larger and more complex ones. To address this limitation, we propose a combination of retrieval and fine-tuning: we can adapt the transformer to a local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours in context. Using TabPFN as the base model -- currently the best tabular in-context learner -- and applying our retrieval and fine-tuning scheme on top results in what we call a locally-calibrated PFN, or LoCalPFN. We conduct extensive evaluation on 95 datasets curated by TabZilla from OpenML, upon which we establish a new state-of-the-art with LoCalPFN -- even with respect to tuned tree-based models. Notably, we show a significant boost in performance compared to the base in-context model, demonstrating the efficacy of our approach and advancing the frontier of deep learning in tabular data.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

38 of 57 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
26 more
  1. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  2. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  3. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  4. 2024The Faiss librarycited by 370
  5. 2021Transformers Can Do Bayesian Inferencecited by 338
  6. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  7. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  8. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  9. 2015Semi-supervised Sequence Learningcited by 198
  10. 2017OpenML Benchmarking Suitescited by 169
  11. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  12. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  13. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  14. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  15. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
  16. 2020Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Datacited by 52
  17. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  18. 2020Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Datacited by 37
  19. 2024Mixture of In-Context Prompters for Tabular PFNscited by 32
  20. 2021Retrieval & Interaction Machine for Tabular Data Predictioncited by 21
  21. 2024In-Context Data Distillation with TabPFNcited by 19
  22. 2024Elephants Never Forget: Testing Language Models for Memorization of Tabular Datacited by 19
  23. 2020Google Dataset Search by the Numberscited by 17
  24. 2022DNNR: Differential Nearest Neighbors Regressioncited by 11
  25. 2024Interpretable Machine Learning for TabPFNcited by 10
  26. 2023Fine-Tuning the Retrieval Mechanism for Tabular Deep Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 24; 2026: 31.

58 papers.

Sort:

By area:

  1. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  2. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  3. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  4. 2024Retrieval Augmented Time Series Forecastingcited by 40
  5. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  6. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  7. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  8. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 28
  9. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 19
  10. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 15
  11. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 15
  12. 2025Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Modelscited by 13
38 more
  1. 2025TabFlex: Scaling Tabular Learning to Millions with Linear Attentioncited by 13
  2. 2025On Finetuning Tabular Foundation Modelscited by 13
  3. 2026Exploring Fine-Tuning for Tabular Foundation Modelscited by 9
  4. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  5. 2025Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Datacited by 7
  6. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  7. 2026TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attentioncited by 5
  8. 2025Uncertainty Quantification for Prior-Data Fitted Networks using Martingale Posteriorscited by 4
  9. 2026TabSieve: Explicit In-Table Evidence Selection for Tabular Predictioncited by 4
  10. 2026Baguan-TS: A Sequence-Native In-Context Learning Model for Time Series Forecasting with Covariatescited by 4
  11. 2026TabDPT-Turbo: Efficient In-Context Learning for Tabular Predictioncited by 4
  12. 2025Realistic Evaluation of TabPFN v2 in Open Environmentscited by 3
  13. 2025Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?cited by 3
  14. 2025TiCT: A Synthetically Pre-Trained Foundation Model for Time Series Classificationcited by 3
  15. 2026Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Modelscited by 3
  16. 2026Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustmentcited by 3
  17. 2026VIP-COP: Context Optimization for Tabular Foundation Modelscited by 3
  18. 2025Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboardcited by 2
  19. 2025Cross-Question Method Reuse in Large Language Models: From Word-Level Prediction to Rational Logical-Layer Reasoningcited by 2
  20. 2026Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraintscited by 2
  21. 2026CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batchingcited by 2
  22. 2026Understanding the Surprising Generalization Properties of Tabular Foundation Modelscited by 2
  23. 2025Enabling Few-Shot Alzheimer's Disease Diagnosis on Biomarker Data with Tabular LLMscited by 1
  24. 2025Uncertainty-Guided Model Selection for Tabular Foundation Models in Biomolecule Efficacy Predictioncited by 1
  25. 2025iLTM: Integrated Large Tabular Modelcited by 1
  26. 2026Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networkscited by 1
  27. 2026Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillationcited by 1
  28. 2026Balanced Adaptive Prototype Selection for Scalable TabPFN Inference on Large-Scale Tabular Datacited by 1
  29. 2026ARASH: Adaptive Retrieval And Shot Selection for Tabular Predictioncited by 1
  30. 2026GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Modelscited by 1
  31. 2026SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learningcited by 1
  32. 2025GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?cited by 0
  33. 2026Tabular LLMs for Interpretable Few-Shot Alzheimer's Disease Prediction with Multimodal Biomedical Datacited by 0
  34. 2026TabCF: Distributional Control Function Estimation with Tabular Foundation Modelscited by 0
  35. 2026Algorithmic Recourse of In-Context Learning for Tabular Datacited by 0
  36. 2026Data-efficient flood depth prediction through domain-aware coreset selection and tabular foundation modelscited by 0
  37. 2026KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Modelscited by 0
  38. 2026Localized TabICLv2: Scaling Tabular In-Context Learning through k-NNcited by 0

50 most cited of 58 All 58 (JSON)