# Foundation Models for Credit Risk Prediction: A Game Changer?

arXiv: https://arxiv.org/abs/2605.18147 · HTML: https://mlsanity.com/paper/2605.18147/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 104 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [Deep Learning for Time Series Forecasting: Tutorial and Literature Survey](https://mlsanity.com/paper/2004.10240/) (2020)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning](https://mlsanity.com/paper/2008.09202/) (2020)
- [Fairness in Credit Scoring: Assessment, Implementation and Profit Implications](https://mlsanity.com/paper/2103.01907/) (2021)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [The Value of Big Data for Credit Scoring: Enhancing Financial Inclusion using Mobile Phone Data and Social Network Analytics](https://mlsanity.com/paper/2002.09931/) (2020)
- [The value of text for small business default prediction: A deep learning approach](https://mlsanity.com/paper/2003.08964/) (2020)
- [The Fairness of Credit Scoring Models](https://mlsanity.com/paper/2205.10200/) (2022)
- [Crowd, Lending, Machine, and Bias](https://mlsanity.com/paper/2008.04068/) (2020)
- [Multilayer Network Analysis for Improved Credit Risk Prediction](https://mlsanity.com/paper/2010.09559/) (2020)
- [A transformer-based model for default prediction in mid-cap corporate markets](https://mlsanity.com/paper/2111.09902/) (2021)
- [Optimizing Credit Limit Adjustments Under Adversarial Goals Using Reinforcement Learning](https://mlsanity.com/paper/2306.15585/) (2023)
- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [Algorithmic decision making methods for fair credit scoring](https://mlsanity.com/paper/2209.07912/) (2022)
- [The Deep Promotion Time Cure Model](https://mlsanity.com/paper/2305.11575/) (2023)

## Built on by (most cited first, 3 of 3)

- [textttfindr: Transparent and Fair Credit Risk Decisions through Semi-Structured Regressions](https://mlsanity.com/paper/2608.24582/) (2026)
- [Is TabPFN the Silver Bullet for Insurance Pricing?](https://mlsanity.com/paper/2605.22892/) (2026)
- [PaMIR: Open Benchmark of Public Credit-Default Datasets](https://mlsanity.com/paper/2610.03259/) (2026)

## Abstract (arXiv)

Predictive models play a pivotal role in credit risk management, guiding critical decisions through accurate estimation of default probabilities and losses. Extensive research has introduced new modeling techniques, complemented by large-scale benchmarking studies consolidating the state-of-the-art. Today, quasi-standards such as gradient-boosting models paired with SHAP explainers have emerged, yet continuous improvement of risk models remains a top priority. Concurrently, rapid advancements in AI, most notably large language models, have disrupted predictive modeling paradigms. Foundation models, pretrained on extensive datasets from diverse domains, have demonstrated remarkable performance by leveraging prior knowledge. While prevalent in natural language processing and computer vision, foundation models for tabular data have only recently emerged. We conjecture that pretraining on out-of-domain data is particularly beneficial in small-data settings, such as SME lending or specialized corporate portfolios, and may help address longstanding challenges including low default portfolios and class imbalance. This paper benchmarks recently proposed tabular foundation models against a broad set of competitors, including established and advanced machine learning techniques, across two core tasks: PD and LGD modeling. Our evaluation encompasses various datasets, performance indicators, and experimental conditions. We find that tabular foundation models generally perform best across datasets and tasks. Moreover, they offer significant improvement in predictive performance as dataset size shrinks. These results are remarkable given that the models are tested out-of-the-box, without hyperparameter tuning, ensuring ease of use and mitigating computational costs.