# Multimodal Document Analytics for Banking Process Automation

arXiv: https://arxiv.org/abs/2307.11845 · HTML: https://mlsanity.com/paper/2307.11845/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 75 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [DiT: Self-supervised Pre-training for Document Image Transformer](https://mlsanity.com/paper/2203.02378/) (2022)
- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://mlsanity.com/paper/2104.08836/) (2021)
- [MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding](https://mlsanity.com/paper/2110.08518/) (2021)
- [The value of text for small business default prediction: A deep learning approach](https://mlsanity.com/paper/2003.08964/) (2020)
- [Multimodal Deep Learning for Finance: Integrating and Forecasting International Stock Markets](https://mlsanity.com/paper/1903.06478/) (2019)
- [Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams](https://mlsanity.com/paper/2304.10740/) (2023)
- [XDoc: Unified Pre-training for Cross-Format Document Understanding](https://mlsanity.com/paper/2210.02849/) (2022)
- [Table Detection for Visually Rich Document Images](https://mlsanity.com/paper/2305.19181/) (2023)
- [NLP in FinTech Applications: Past, Present and Future](https://mlsanity.com/paper/2005.01320/) (2020)
- [Company2Vec -- German Company Embeddings based on Corporate Websites](https://mlsanity.com/paper/2307.09332/) (2023)

## Built on by (most cited first, 2 of 2)

- [mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning](https://mlsanity.com/paper/2404.01548/) (2024)
- [SynJAC: Synthetic-data-driven Joint-granular Adaptation and Calibration for Domain Specific Scanned Document Key Information Extraction](https://mlsanity.com/paper/2410.01609/) (2024)

## Abstract (arXiv)

Traditional banks face increasing competition from FinTechs in the rapidly evolving financial ecosystem. Raising operational efficiency is vital to address this challenge. Our study aims to improve the efficiency of document-intensive business processes in banking. To that end, we first review the landscape of business documents in the retail segment. Banking documents often contain text, layout, and visuals, suggesting that document analytics and process automation require more than plain natural language processing (NLP). To verify this and assess the incremental value of visual cues when processing business documents, we compare a recently proposed multimodal model called LayoutXLM to powerful text classifiers (e.g., BERT) and large language models (e.g., GPT) in a case study related to processing company register extracts. The results confirm that incorporating layout information in a model substantially increases its performance. Interestingly, we also observed that more than 75% of the best model performance (in terms of the F1 score) can be achieved with as little as 30% of the training data. This shows that the demand for data labeled data to set up a multi-modal model can be moderate, which simplifies real-world applications of multimodal document analytics. Our study also sheds light on more specific practices in the scope of calibrating a multimodal banking document classifier, including the need for fine-tuning. In sum, the paper contributes original empirical evidence on the effectiveness and efficiency of multi-model models for document processing in the banking business and offers practical guidance on how to unlock this potential in day-to-day operations.