mlsanityData to 7 Oct 2026

TabICL: A Tabular Foundation Model for In-Context Learning on Large Data

arXiv 2502.05564 ↗First posted 8 Feb 2025cs.LG cs.AIcited by 267

Not on the map. Why?

Abstract (arXiv)

The long-standing dominance of gradient-boosted decision trees on tabular data is currently challenged by tabular foundation models using In-Context Learning (ICL): setting the training data as context for the test data and predicting in a single forward pass without parameter updates. While TabPFNv2 foundation model excels on tables with up to 10K samples, its alternating column- and row-wise attentions make handling large training sets computationally prohibitive. So, can ICL be effectively scaled and deliver a benefit for larger tables? We introduce TabICL, a tabular foundation model for classification, pretrained on synthetic datasets with up to 60K samples and capable of handling 500K samples on affordable resources. This is enabled by a novel two-stage architecture: a column-then-row attention mechanism to build fixed-dimensional embeddings of rows, followed by a transformer for efficient ICL. Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times), and significantly outperforms all other approaches. On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the potential of ICL for large data. Pretraining code, inference code, and pre-trained models are available at https://github.com/soda-inria/tabicl.

Key facts. Cited by 267 papers in the map (data through 7 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 8 Feb 2025.

Builds on

36 of 53 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2024Chronos: Learning the Language of Time Seriescited by 712
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  12. 2017Self-Normalizing Neural Networkscited by 591
24 more
  1. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  2. 2021Transformers Can Do Bayesian Inferencecited by 338
  3. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  4. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  5. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  6. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  7. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  8. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  9. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
  10. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  11. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
  12. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  13. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  14. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
  15. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 69
  16. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 58
  17. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  18. 2021Adversarial Attacks for Tabular Data: Application to Fraud Detection and Imbalanced Datacited by 45
  19. 2024A Closer Look at Deep Learning Methods on Tabular Datasetscited by 36
  20. 2024Mixture of In-Context Prompters for Tabular PFNscited by 32
  21. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 29
  22. 2024In-Context Data Distillation with TabPFNcited by 19
  23. 2024Elephants Never Forget: Testing Language Models for Memorization of Tabular Datacited by 19
  24. 2023Sample Path Regularity of Gaussian Processes from the Covariance Kernelcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 61; 2026: 206.

267 papers.

Sort:

By area:

  1. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 115
  2. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 67
  3. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  4. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  5. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  6. 2025Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Datacited by 36
  7. 2025Relational Transformer: Toward Zero-Shot Foundation Models for Relational Datacited by 32
  8. 2025ConTextTab: A Semantics-Aware Tabular In-Context Learnercited by 31
  9. 2025Turning Tabular Foundation Models into Graph Foundation Modelscited by 30
  10. 2025Mantis: Lightweight Foundation Model for Time Series Classificationcited by 21
  11. 2025GraphPFN: A Prior-Data Fitted Graph Foundation Modelcited by 20
  12. 2026RelBench v2: A Large-Scale Benchmark and Repository for Relational Datacited by 17
38 more
  1. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 15
  2. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  3. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 15
  4. 2026PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Modelscited by 15
  5. 2026KumoRFM-2: Scaling Foundation Models for Relational Learningcited by 15
  6. 2025xRFM: Accurate, scalable, and interpretable feature learning models for tabular datacited by 14
  7. 2025Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Modelscited by 13
  8. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 13
  9. 2025Bringing Graphs to the Table: Zero-shot Node Classification via Tabular Foundation Modelscited by 12
  10. 2026Relational In-Context Learning via Synthetic Pre-training with Structural Priorcited by 11
  11. 2025TabPFN: One Model to Rule Them All?cited by 10
  12. 2025Early Stopping Tabular In-Context Learningcited by 10
  13. 2025TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Modelscited by 10
  14. 2025nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFNcited by 10
  15. 2026MantisV2: Closing the Zero-Shot Gap in Time Series Classification with Synthetic Data and Test-Time Strategiescited by 10
  16. 2026Exploring Fine-Tuning for Tabular Foundation Modelscited by 9
  17. 2025Towards Fair In-Context Learning with Tabular Foundation Modelscited by 8
  18. 2025Conditional Coverage Diagnostics for Conformal Predictioncited by 8
  19. 2026From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detectioncited by 8
  20. 2026Predictive Query Language: A Domain-Specific Language for Predictive Modeling on Relational Databasescited by 8
  21. 2026No Need to Train Your RDB Foundation Modelcited by 8
  22. 2026EXAONE Tabular 1.0 : Technical Reportcited by 7
  23. 2025ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Datacited by 6
  24. 2025Orion-Bix: Bi-Axial Attention for Tabular In-Context Learningcited by 6
  25. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  26. 2026Use What You Know: Causal Foundation Models with Partial Graphscited by 6
  27. 2026LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Modelscited by 6
  28. 2025Position: Foundation Models for Tabular Data within Systemic Contexts Need Groundingcited by 5
  29. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 5
  30. 2025In-Context Multi-Objective Optimizationcited by 5
  31. 2026End-to-End Compression for Tabular Foundation Modelscited by 5
  32. 2026TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attentioncited by 5
  33. 2025Uncertainty Quantification for Prior-Data Fitted Networks using Martingale Posteriorscited by 4
  34. 2025Efficient Autoregressive Inference for Transformer Probabilistic Modelscited by 4
  35. 2026TabClustPFN: A Prior-Fitted Network for Tabular Data Clusteringcited by 4
  36. 2026TabSieve: Explicit In-Table Evidence Selection for Tabular Predictioncited by 4
  37. 2026TACTIC for Navigating the Unknown: Tabular Anomaly deteCTion via In-Context inferencecited by 4
  38. 2026Tabular foundation models for in-context prediction of molecular propertiescited by 4

50 most cited of 267 All 267 (JSON)