mlsanityData to 14 Oct 2026

LLM Embeddings Improve Test-time Adaptation to Tabular Y|X-Shifts

arXiv 2410.07395 ↗First posted 9 Oct 2024cs.LG cs.AI math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

For tabular datasets, the change in the relationship between the label and covariates (Y|X-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples. We focus on building more informative representations of tabular data that can mitigate Y|X-shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies. The code is available at https://github.com/namkoong-lab/LLM-Tabular-Shifts.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

42 of 62 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2019Invariant Risk Minimizationcited by 1,854
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  7. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  8. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  10. 2021Domain Generalization: A Surveycited by 764
  11. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  12. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
30 more
  1. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  2. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
  3. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  4. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 449
  5. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  6. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  7. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  8. 2023Improving Text Embeddings with Large Language Modelscited by 303
  9. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  10. 2020The Risks of Invariant Risk Minimizationcited by 262
  11. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  12. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  13. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  14. 2016Variance-based regularization with convex objectivescited by 155
  15. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  16. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  17. 2020Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimalitycited by 90
  18. 2021Transfer Learning under High-dimensional Generalized Linear Modelscited by 70
  19. 2019Do Image Classifiers Generalize Across Time?cited by 59
  20. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 57
  21. 2020When is invariance useful in an Out-of-Distribution Generalization problem ?cited by 52
  22. 2023Improved Test-Time Adaptation for Domain Generalizationcited by 47
  23. 2023Diagnosing Model Performance Under Distribution Shiftcited by 39
  24. 2018Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemescited by 30
  25. 2023TABLET: Learning From Instructions For Tabular Datacited by 28
  26. 2023From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Modelscited by 24
  27. 2022Subgroup Robustness Grows On Trees: An Empirical Baseline Investigationcited by 17
  28. 2017Data-driven Optimal Cost Selection for Distributionally Robust Optimizationcited by 15
  29. 2023Unifying Distributionally Robust Optimization via Optimal Transport Theorycited by 13
  30. 2024Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretrainingcited by 12

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  2. 2025LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domainscited by 2
  3. 2025Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddingscited by 0