# WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain

arXiv: https://arxiv.org/abs/2211.00083 · HTML: https://mlsanity.com/paper/2211.00083/

## Key facts

- Cited by 117 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [Span Selection Pre-training for Question Answering](https://mlsanity.com/paper/1909.04120/) (2019)
- [Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation](https://mlsanity.com/paper/2010.02705/) (2020)
- [Financial Aspect-Based Sentiment Analysis using Deep Representations](https://mlsanity.com/paper/1808.07931/) (2018)

## Built on by (most cited first, 30 of 117)

- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://mlsanity.com/paper/2401.11641/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [A Survey of Large Language Models in Finance (FinLLMs)](https://mlsanity.com/paper/2402.02315/) (2024)
- [Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis](https://mlsanity.com/paper/2305.07972/) (2023)
- [FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets](https://mlsanity.com/paper/2310.04793/) (2023)
- [INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent](https://mlsanity.com/paper/2412.18174/) (2024)
- [TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge](https://mlsanity.com/paper/2310.15051/) (2023)
- [BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark](https://mlsanity.com/paper/2302.09432/) (2023)
- [Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks](https://mlsanity.com/paper/2305.05862/) (2023)
- [Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models](https://mlsanity.com/paper/2306.12659/) (2023)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.10708/) (2025)
- [Designing Heterogeneous LLM Agents for Financial Sentiment Analysis](https://mlsanity.com/paper/2401.05799/) (2024)
- [Financial Report Chunking for Effective Retrieval Augmented Generation](https://mlsanity.com/paper/2402.05131/) (2024)
- [CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering](https://mlsanity.com/paper/2403.16008/) (2024)
- [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025)
- [MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining](https://mlsanity.com/paper/2312.17482/) (2023)
- [OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain](https://mlsanity.com/paper/2412.13018/) (2024)
- [STEER: Assessing the Economic Rationality of Large Language Models](https://mlsanity.com/paper/2402.09552/) (2024)
- [Simulating Financial Market via Large Language Model based Agents](https://mlsanity.com/paper/2406.19966/) (2024)
- [Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models](https://mlsanity.com/paper/2310.00566/) (2023)
- [ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks](https://mlsanity.com/paper/2407.06245/) (2024)
- [Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset](https://mlsanity.com/paper/2405.10542/) (2024)

## Abstract (arXiv)

Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data are publicly available on Github and Huggingface.