mlsanityData to 14 Oct 2026

PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance

arXiv 2306.05443 ↗First posted 8 Jun 2023cs.CL cs.AIcited by 222

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Although large language models (LLMs) has shown great performance on natural language processing (NLP) in the financial domain, there are no publicly available financial tailtored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (AI). This paper introduces PIXIU, a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 136K data samples to support the fine-tuning, and an evaluation benchmark with 5 tasks and 9 datasets. We first construct the large-scale multi-task instruction data considering a variety of financial tasks, financial document types, and financial data modalities. We then propose a financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including five financial NLP tasks and one financial prediction task. With this benchmark, we conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced to facilitate future research in financial AI.

Key facts. Cited by 222 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

36 of 50 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
24 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2023BloombergGPT: A Large Language Model for Financecited by 939
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  6. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  7. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  8. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  9. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  10. 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
  11. 2023Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Modelscited by 129
  12. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  13. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  14. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  15. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 75
  16. 2023BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmarkcited by 60
  17. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  18. 2023The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challengescited by 59
  19. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  20. 2021Trade the Event: Corporate Events Detection for News-Based Event-Driven Tradingcited by 39
  21. 2023FiNER-ORD: Financial Named Entity Recognition Open Research Datasetcited by 17
  22. 2023Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Taskscited by 14
  23. 2023Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learningcited by 6
  24. 2023When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLPcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 23; 2024: 65; 2025: 74; 2026: 60.

222 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  4. 2023Large Language Models in Finance: A Surveycited by 226
  5. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  6. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 141
  7. 2024Large Language Models for Time Series: A Surveycited by 114
  8. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  9. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 90
  10. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 90
  11. 2023InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuningcited by 82
  12. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 79
38 more
  1. 2023FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasetscited by 69
  2. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 67
  3. 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
  4. 2024INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agentcited by 64
  5. 2023MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Modelscited by 63
  6. 2024EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysiscited by 62
  7. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 55
  8. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  9. 2024Position: What Can Large Language Models Tell Us about Time Series Analysiscited by 47
  10. 2024AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Frameworkcited by 47
  11. 2023Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucinationcited by 41
  12. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  13. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 38
  14. 2024Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Modelscited by 36
  15. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 34
  16. 2024AI for social science and social science of AI: A Surveycited by 29
  17. 2024A Survey of Useful LLM Evaluationcited by 29
  18. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 29
  19. 2023BizBench: A Quantitative Reasoning Benchmark for Business and Financecited by 28
  20. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 28
  21. 2024Simulating Financial Market via Large Language Model based Agentscited by 25
  22. 2023Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Modelscited by 24
  23. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  24. 2025FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Tradingcited by 23
  25. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMscited by 22
  26. 2025LLM-Powered Multi-Agent System for Automated Crypto Portfolio Managementcited by 20
  27. 2023CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Modelcited by 18
  28. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  29. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  30. 2025Plutus: Benchmarking Large Language Models in Low-Resource Greek Financecited by 18
  31. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  32. 2024Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoningcited by 16
  33. 2026FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answeringcited by 16
  34. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  35. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  36. 2025Trading-R1: Financial Trading with LLM Reasoning via Reinforcement Learningcited by 15
  37. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  38. 2025FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMscited by 14

50 most cited of 222 All 222 (JSON)