mlsanityData to 7 Oct 2026

Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training

arXiv 2404.10555 ↗First posted 16 Apr 2024cs.CL q-fin.CPcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are now widely used in various fields, including finance. However, Japanese financial-specific LLMs have not been proposed yet. Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training. Before tuning, we constructed Japanese financial-focused datasets for continual pre-training. As a base model, we employed a Japanese LLM that achieved state-of-the-art performance on Japanese financial benchmarks among the 10-billion-class parameter models. After continual pre-training using the datasets and the base model, the tuned model performed better than the original model on the Japanese financial benchmarks. Moreover, the outputs comparison results reveal that the tuned model's outputs tend to be better than the original model's outputs in terms of the quality and length of the answers. These findings indicate that domain-specific continual pre-training is also effective for LLMs. The tuned model is publicly available on Hugging Face.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

27 of 43 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
15 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  9. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  10. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  11. 2023Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Modelscited by 58
  12. 2024Construction of a Japanese Financial Benchmark for Large Language Modelscited by 12
  13. 2023llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodologycited by 6
  14. 2023JMedLoRA:Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuningcited by 6
  15. 2023From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Modelscited by 2
  2. 2026Ebisu: Benchmarking Large Language Models in Japanese Financecited by 1
  3. 2026Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Trainingcited by 0