# Construction of a Japanese Financial Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2403.15062 · HTML: https://mlsanity.com/paper/2403.15062/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology](https://mlsanity.com/paper/2305.12720/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering](https://mlsanity.com/paper/2503.16858/) (2025)
- [MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application](https://mlsanity.com/paper/2506.14028/) (2025)
- [Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk](https://mlsanity.com/paper/2502.15865/) (2025)
- [SoK: Security of Autonomous LLM Agents in Agentic Commerce](https://mlsanity.com/paper/2604.15367/) (2026)
- [EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements](https://mlsanity.com/paper/2506.08762/) (2025)
- [Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training](https://mlsanity.com/paper/2404.10555/) (2024)
- [The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging](https://mlsanity.com/paper/2409.19854/) (2024)
- [JaFIn: Japanese Financial Instruction Dataset](https://mlsanity.com/paper/2404.09260/) (2024)
- [Economy Watchers Survey Provides Datasets and Tasks for Japanese Financial Domain](https://mlsanity.com/paper/2407.14727/) (2024)
- [Refined and Segmented Price Sentiment Indices from Survey Comments](https://mlsanity.com/paper/2411.09937/) (2024)
- [JFinTEB: Japanese Financial Text Embedding Benchmark](https://mlsanity.com/paper/2604.15882/) (2026)
- [Fin_Agent at FinEval 2026: Task-Aware Methods for Multilingual Financial NLP](https://mlsanity.com/paper/2610.14101/) (2026)

## Abstract (arXiv)

With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity. There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain. Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models. Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively. According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.