# XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters

arXiv: https://arxiv.org/abs/2305.12002 · HTML: https://mlsanity.com/paper/2305.12002/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark](https://mlsanity.com/paper/2302.09432/) (2023)
- [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://mlsanity.com/paper/2110.06696/) (2021)

## Built on by (most cited first, 30 of 92)

- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [Data-Juicer: A One-Stop Data Processing System for Large Language Models](https://mlsanity.com/paper/2309.02033/) (2023)
- [Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.10708/) (2025)
- [EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce](https://mlsanity.com/paper/2308.06966/) (2023)
- [TrafficGPT: Viewing, Processing and Interacting with Traffic Foundation Models](https://mlsanity.com/paper/2309.06719/) (2023)
- [Baichuan-M1: Pushing the Medical Capability of Large Language Models](https://mlsanity.com/paper/2502.12671/) (2025)
- [TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models](https://mlsanity.com/paper/2407.09424/) (2024)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs](https://mlsanity.com/paper/2404.07584/) (2024)
- [Align on the Fly: Adapting Chatbot Behavior to Established Norms](https://mlsanity.com/paper/2312.15907/) (2023)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation](https://mlsanity.com/paper/2312.17276/) (2023)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [ChIRAAG: ChatGPT Informed Rapid and Automated Assertion Generation](https://mlsanity.com/paper/2402.00093/) (2024)

## Abstract (arXiv)

In recent years, pre-trained language models have undergone rapid development with the emergence of large-scale models. However, there is a lack of open-sourced chat models specifically designed for the Chinese language, especially in the field of Chinese finance, at the scale of hundreds of billions. To address this gap, we introduce XuanYuan 2.0, the largest Chinese chat model to date, built upon the BLOOM-176B architecture. Additionally, we propose a novel training method called hybrid-tuning to mitigate catastrophic forgetting. By combining general-domain with domain-specific knowledge and integrating the stages of pre-training and fine-tuning, XuanYuan 2.0 is capable of providing accurate and contextually appropriate responses in the Chinese financial domain.