# A transformer-based model for default prediction in mid-cap corporate markets

arXiv: https://arxiv.org/abs/2111.09902 · HTML: https://mlsanity.com/paper/2111.09902/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 65 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series](https://mlsanity.com/paper/1811.10166/) (2018)
- [Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks](https://mlsanity.com/paper/1510.00562/) (2015)
- [A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation](https://mlsanity.com/paper/1806.06957/) (2018)
- [The value of text for small business default prediction: A deep learning approach](https://mlsanity.com/paper/2003.08964/) (2020)

## Built on by (most cited first, 6 of 6)

- [Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams](https://mlsanity.com/paper/2304.10740/) (2023)
- [The Deep Promotion Time Cure Model](https://mlsanity.com/paper/2305.11575/) (2023)
- [Foundation Models for Credit Risk Prediction: A Game Changer?](https://mlsanity.com/paper/2605.18147/) (2026)
- [Large-scale Time-Varying Portfolio Optimisation using Graph Attention Networks](https://mlsanity.com/paper/2407.15532/) (2024)
- [textttfindr: Transparent and Fair Credit Risk Decisions through Semi-Structured Regressions](https://mlsanity.com/paper/2608.24582/) (2026)
- [A Multimodal Approach to SME Credit Scoring Integrating Transaction and Ownership Networks](https://mlsanity.com/paper/2510.09407/) (2025)

## Abstract (arXiv)

In this paper, we study mid-cap companies, i.e. publicly traded companies with less than US 10 billion in market capitalisation. Using a large dataset of US mid-cap companies observed over 30 years, we look to predict the default probability term structure over the medium term and understand which data sources (i.e. fundamental, market or pricing data) contribute most to the default risk. Whereas existing methods typically require that data from different time periods are first aggregated and turned into cross-sectional features, we frame the problem as a multi-label time-series classification problem. We adapt transformer models, a state-of-the-art deep learning model emanating from the natural language processing domain, to the credit risk modelling setting. We also interpret the predictions of these models using attention heat maps. To optimise the model further, we present a custom loss function for multi-label classification and a novel multi-channel architecture with differential training that gives the model the ability to use all input data efficiently. Our results show the proposed deep learning architecture's superior performance, resulting in a 13% improvement in AUC (Area Under the receiver operating characteristic Curve) over traditional models. We also demonstrate how to produce an importance ranking for the different data sources and the temporal relationships using a Shapley approach specific to these models.