mlsanityData to 14 Oct 2026

Construction of a Japanese Financial Benchmark for Large Language Models

arXiv 2403.15062 ↗First posted 22 Mar 2024q-fin.CP cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity. There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain. Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models. Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively. According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 15 of its 39 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

15 of 39 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  12. 2023BloombergGPT: A Large Language Model for Financecited by 939
3 more
  1. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  2. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  3. 2023llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodologycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 90
  2. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 55
  3. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  4. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  5. 2026SoK: Security of Autonomous LLM Agents in Agentic Commercecited by 8
  6. 2025EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statementscited by 6
  7. 2024Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-trainingcited by 3
  8. 2024The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Mergingcited by 3
  9. 2024JaFIn: Japanese Financial Instruction Datasetcited by 2
  10. 2024Economy Watchers Survey Provides Datasets and Tasks for Japanese Financial Domaincited by 2
  11. 2024Refined and Segmented Price Sentiment Indices from Survey Commentscited by 1
  12. 2026JFinTEB: Japanese Financial Text Embedding Benchmarkcited by 0
1 more
  1. 2026Fin_Agent at FinEval 2026: Task-Aware Methods for Multilingual Financial NLPcited by 0