# llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology

arXiv: https://arxiv.org/abs/2305.12720 · HTML: https://mlsanity.com/paper/2305.12720/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [JESC: Japanese-English Subtitle Corpus](https://mlsanity.com/paper/1710.10639/) (2017)
- [Coursera Corpus Mining and Multistage Fine-Tuning for Improving Lectures Translation](https://mlsanity.com/paper/1912.11739/) (2019)

## Built on by (most cited first, 6 of 6)

- [Construction of a Japanese Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2403.15062/) (2024)
- [From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models](https://mlsanity.com/paper/2309.03412/) (2023)
- [MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs](https://mlsanity.com/paper/2505.19800/) (2025)
- [Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training](https://mlsanity.com/paper/2404.10555/) (2024)
- [The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging](https://mlsanity.com/paper/2409.19854/) (2024)
- [JaFIn: Japanese Financial Instruction Dataset](https://mlsanity.com/paper/2404.09260/) (2024)

## Abstract (arXiv)

This study constructed a Japanese chat dataset for tuning large language models (LLMs), which consist of about 8.4 million records. Recently, LLMs have been developed and gaining popularity. However, high-performing LLMs are usually mainly for English. There are two ways to support languages other than English by those LLMs: constructing LLMs from scratch or tuning existing models. However, in both ways, datasets are necessary parts. In this study, we focused on supporting Japanese in those LLMs and making a dataset for training or tuning LLMs in Japanese. The dataset we constructed consisted of various tasks, such as translation and knowledge tasks. In our experiment, we tuned an existing LLM using our dataset and evaluated the performance qualitatively. The results suggest that our dataset is possibly beneficial for LLMs. However, we also revealed some difficulties in constructing LLMs in languages other than English.