# Class-Imbalanced-Aware Adaptive Dataset Distillation for Scalable Pretrained Model on Credit Scoring

arXiv: https://arxiv.org/abs/2501.10677 · HTML: https://mlsanity.com/paper/2501.10677/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 62 references that are in the map.
- First posted on arXiv 18 Jan 2025.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://mlsanity.com/paper/2401.11641/) (2024)
- [A Broad Study of Pre-training for Domain Generalization and Adaptation](https://mlsanity.com/paper/2203.11819/) (2022)
- [Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation](https://mlsanity.com/paper/2209.04599/) (2022)
- [Mixture of In-Context Prompters for Tabular PFNs](https://mlsanity.com/paper/2405.16156/) (2024)
- [A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts](https://mlsanity.com/paper/2001.05326/) (2020)
- [Explainable AI for Interpretable Credit Scoring](https://mlsanity.com/paper/2012.03749/) (2020)
- [Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks](https://mlsanity.com/paper/2311.10609/) (2023)
- [A Novel Ensemble Deep Learning Model for Stock Prediction Based on Stock Prices and News](https://mlsanity.com/paper/2007.12620/) (2020)
- [Imbalance-XGBoost: Leveraging Weighted and Focal Losses for Binary Label-Imbalanced Classification with XGBoost](https://mlsanity.com/paper/1908.01672/) (2019)

## Built on by (most cited first, 1 of 1)

- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)

## Abstract (arXiv)

The advent of artificial intelligence has significantly enhanced credit scoring technologies. Despite the remarkable efficacy of advanced deep learning models, mainstream adoption continues to favor tree-structured models due to their robust predictive performance on tabular data. Although pretrained models have seen considerable development, their application within the financial realm predominantly revolves around question-answering tasks and the use of such models for tabular-structured credit scoring datasets remains largely unexplored. Tabular-oriented large models, such as TabPFN, has made the application of large models in credit scoring feasible, albeit can only processing with limited sample sizes. This paper provides a novel framework to combine tabular-tailored dataset distillation technique with the pretrained model, empowers the scalability for TabPFN. Furthermore, though class imbalance distribution is the common nature in financial datasets, its influence during dataset distillation has not been explored. We thus integrate the imbalance-aware techniques during dataset distillation, resulting in improved performance in financial datasets (e.g., a 2.5% enhancement in AUC). This study presents a novel framework for scaling up the application of large pretrained models on financial tabular datasets and offers a comparative analysis of the influence of class imbalance on the dataset distillation process. We believe this approach can broaden the applications and downstream tasks of large models in the financial domain.