# Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams

arXiv: https://arxiv.org/abs/2304.10740 · HTML: https://mlsanity.com/paper/2304.10740/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 161 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities](https://mlsanity.com/paper/2111.06420/) (2021)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [Deep Learning for Financial Applications : A Survey](https://mlsanity.com/paper/2002.05786/) (2020)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [A Primer on Neural Network Models for Natural Language Processing](https://mlsanity.com/paper/1510.00726/) (2015)
- [Spatial-Temporal Recurrent Neural Network for Emotion Recognition](https://mlsanity.com/paper/1705.04515/) (2017)
- [Language Model is All You Need: Natural Language Understanding as Question Answering](https://mlsanity.com/paper/2011.03023/) (2020)
- [Evaluation of deep learning models for multi-step ahead time series prediction](https://mlsanity.com/paper/2103.14250/) (2021)
- [The Value of Big Data for Credit Scoring: Enhancing Financial Inclusion using Mobile Phone Data and Social Network Analytics](https://mlsanity.com/paper/2002.09931/) (2020)
- [The value of text for small business default prediction: A deep learning approach](https://mlsanity.com/paper/2003.08964/) (2020)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [COVID-19 sentiment analysis via deep learning during the rise of novel cases](https://mlsanity.com/paper/2104.10662/) (2021)

## Built on by (most cited first, 13 of 13)

- [Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models](https://mlsanity.com/paper/2310.00566/) (2023)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning](https://mlsanity.com/paper/2506.13705/) (2025)
- [Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs](https://mlsanity.com/paper/2407.17624/) (2024)
- [Multimodal Document Analytics for Banking Process Automation](https://mlsanity.com/paper/2307.11845/) (2023)
- [Review of deep learning models for crypto price prediction: implementation and evaluation](https://mlsanity.com/paper/2405.11431/) (2024)
- [SenTSR-Bench: Thinking with Injected Knowledge for Time-Series Reasoning](https://mlsanity.com/paper/2602.19455/) (2026)
- [Longitudinal Abuse and Sentiment Analysis of Hollywood Movie Dialogues using Language Models](https://mlsanity.com/paper/2501.13948/) (2025)
- [Data-Local Autonomous LLM-Guided Neural Architecture Search for Multiclass Multimodal Time-Series Classification](https://mlsanity.com/paper/2603.15939/) (2026)
- [Cross-Modal Temporal Fusion for Financial Market Forecasting](https://mlsanity.com/paper/2504.13522/) (2025)
- [CreditARF: A Framework for Corporate Credit Rating with Annual Report and Financial Feature Integration](https://mlsanity.com/paper/2508.02738/) (2025)
- [Why Bonds Fail Differently? Explainable Multimodal Learning for Multi-Class Default Prediction](https://mlsanity.com/paper/2509.10802/) (2025)
- [A Multimodal Approach to SME Credit Scoring Integrating Transaction and Ownership Networks](https://mlsanity.com/paper/2510.09407/) (2025)

## Abstract (arXiv)

Knowing which factors are significant in credit rating assignment leads to better decision-making. However, the focus of the literature thus far has been mostly on structured data, and fewer studies have addressed unstructured or multi-modal datasets. In this paper, we present an analysis of the most effective architectures for the fusion of deep learning models for the prediction of company credit rating classes, by using structured and unstructured datasets of different types. In these models, we tested different combinations of fusion strategies with different deep learning models, including CNN, LSTM, GRU, and BERT. We studied data fusion strategies in terms of level (including early and intermediate fusion) and techniques (including concatenation and cross-attention). Our results show that a CNN-based multi-modal model with two fusion strategies outperformed other multi-modal techniques. In addition, by comparing simple architectures with more complex ones, we found that more sophisticated deep learning models do not necessarily produce the highest performance; however, if attention-based models are producing the best results, cross-attention is necessary as a fusion strategy. Finally, our comparison of rating agencies on short-, medium-, and long-term performance shows that Moody's credit ratings outperform those of other agencies like Standard & Poor's and Fitch Ratings.