mlsanityData to 7 Oct 2026

In-context Learning of Evolving Data Streams with Tabular Foundational Models

arXiv 2502.16840 ↗First posted 24 Feb 2025cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

State-of-the-art data stream mining has long drawn from ensembles of the Very Fast Decision Tree, a seminal algorithm honored with the 2015 KDD Test-of-Time Award. However, the emergence of large tabular models, i.e., transformers designed for structured numerical data, marks a significant paradigm shift. These models move beyond traditional weight updates, instead employing in-context learning through prompt tuning. By using on-the-fly sketches to summarize unbounded streaming data, one can feed this information into a pre-trained model for efficient processing. This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments. Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees, such as Adaptive Random Forest, and Streaming Random Patches, across all non-stationary benchmarks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 74 of its 145 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

74 of 145 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2016HyperNetworkscited by 1,003
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  12. 2022Transformers learn in-context by gradient descentcited by 714
38 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  3. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  4. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  5. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  6. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  7. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  8. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  9. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  10. 2021Transformers Can Do Bayesian Inferencecited by 338
  11. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  13. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
  14. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  15. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  16. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  17. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  18. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  19. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  20. 2022Binding Language Models in Symbolic Languagescited by 224
  21. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  22. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  23. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  24. 2022Can Foundation Models Wrangle Your Data?cited by 173
  25. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  26. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  27. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  28. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  29. 2017Few-Shot Image Recognition by Predicting Parameters from Activationscited by 122
  30. 2021TABBIE: Pretrained Representations of Tabular Datacited by 111
  31. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  32. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  33. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  34. 2020TUTA: Tree-based Transformers for Generally Structured Table Pre-trainingcited by 100
  35. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  36. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  37. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  38. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 115
  2. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  3. 2026Bounded Context Management for Tabular Foundation Models on Stream Learningcited by 1
  4. 2026Streaming Hierarchical Inference with Tabular Foundation Modelscited by 1
  5. 2026NOMADD: Numerical Optimization of Models Adapting to Data Driftcited by 0
  6. 2026Collaborative Streaming Anomaly Detection with Interactive Explanations and Ensemble Consensuscited by 0
  7. 2026Do Existing Preconditioners Improve Biomedical Tabular Foundation Learning? An Empirical Study on TabPFN Optimizationcited by 0