mlsanityData to 14 Oct 2026

Mixture of In-Context Prompters for Tabular PFNs

arXiv 2405.16156 ↗First posted 25 May 2024cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Recent benchmarks found In-Context Learning (ICL) outperforms both deep learning and tree-based algorithms on small tabular datasets. However, on larger datasets, ICL for tabular learning cannot run without severely compromising performance, due to its quadratic space and time complexity w.r.t. dataset size. We propose MIXTUREPFN, which both extends nearest-neighbor sampling to the state-of-the-art ICL for tabular learning model and uses bootstrapping to finetune said model on the inference-time dataset. MIXTUREPFN is the Condorcet winner across 36 diverse tabular datasets against 19 strong deep learning and tree-based baselines, achieving the highest mean rank among Top-10 aforementioned algorithms with statistical significance.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 47 of its 67 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

47 of 67 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
35 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  8. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  9. 2022Transformers learn in-context by gradient descentcited by 732
  10. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  11. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  12. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  13. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  14. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  15. 2021Transformers Can Do Bayesian Inferencecited by 360
  16. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  17. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  18. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  19. 2021Hash Layers For Large Sparse Modelscited by 237
  20. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  21. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  22. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  23. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  24. 2018Meta-Learning Probabilistic Inference For Predictioncited by 110
  25. 2022The Devil in Linear Transformercited by 98
  26. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  27. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  28. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  29. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  30. 2021DANets: Deep Abstract Networks for Tabular Data Classification and Regressioncited by 50
  31. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  32. 2023Pre-Training to Learn in Contextcited by 42
  33. 2022Hopular: Modern Hopfield Networks for Tabular Datacited by 18
  34. 2023Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networkscited by 18
  35. 2022Introducing Semantics into Speech Encoderscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 14.

33 papers.

Sort:

By area:

  1. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  2. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  3. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  4. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  5. 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 51
  6. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  7. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 36
  8. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  9. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 20
  10. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 16
  11. 2025TabFlex: Scaling Tabular Learning to Millions with Linear Attentioncited by 15
  12. 2025Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Modelscited by 13
21 more
  1. 2025On Finetuning Tabular Foundation Modelscited by 13
  2. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 8
  3. 2025Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Datacited by 8
  4. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 7
  5. 2025Realistic Evaluation of TabPFN v2 in Open Environmentscited by 4
  6. 2026Baguan-TS: A Sequence-Native In-Context Learning Model for Time Series Forecasting with Covariatescited by 4
  7. 2026Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraintscited by 4
  8. 2026Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustmentcited by 4
  9. 2026Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Modelscited by 3
  10. 2026VIP-COP: Context Optimization for Tabular Foundation Modelscited by 3
  11. 2026CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batchingcited by 2
  12. 2025Class-Imbalanced-Aware Adaptive Dataset Distillation for Scalable Pretrained Model on Credit Scoringcited by 1
  13. 2025Iceberg: Enhancing HLS Modeling with Synthetic Datacited by 1
  14. 2026GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Modelscited by 1
  15. 2026LoGIC: Budgeted Context Construction for Node-Level Graph In-Context Learning with Tabular Foundation Modelscited by 1
  16. 2026TabSurv: Adapting Modern Tabular Neural Networks to Survival Analysiscited by 0
  17. 2026Parameter-Efficient Distributionally Robust Adaptation of Tabular Foundation Models under Subpopulation Shiftcited by 0
  18. 2026Distillation of Tabular Foundation Models into Efficient Predictorscited by 0
  19. 2026Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learningcited by 0
  20. 2026Test-Time Compute for Tabular Foundation Models: Mechanisms, Gains, and Limitscited by 0
  21. 2026How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?cited by 0