mlsanityData to 7 Oct 2026

From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models

arXiv 2309.03412 ↗First posted 7 Sep 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well verified in non-English languages. We construct a Japanese instruction dataset by expanding and filtering existing datasets and apply the dataset to a Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models using our instruction dataset. We evaluated these models from both quantitative and qualitative perspectives. As a result, the effectiveness of Japanese instruction datasets is confirmed. The results also indicate that even with relatively small LLMs, performances in downstream tasks would be improved through instruction tuning. Our instruction dataset, tuned models, and implementation are publicly available online.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 40 of its 58 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

40 of 58 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
28 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  13. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2023LIMA: Less Is More for Alignmentcited by 1,080
  16. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  17. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  18. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  19. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  20. 2023Instruction Tuning with GPT-4cited by 676
  21. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  22. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  23. 2022UL2: Unifying Language Learning Paradigmscited by 292
  24. 2020The Multilingual Amazon Reviews Corpuscited by 142
  25. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  26. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  27. 2023Chinese Open Instruction Generalist: A Preliminary Releasecited by 35
  28. 2023llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodologycited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2023JMedLoRA:Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuningcited by 6
  2. 2024Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-trainingcited by 3
  3. 2024JaFIn: Japanese Financial Instruction Datasetcited by 2
  4. 2024Refined and Segmented Price Sentiment Indices from Survey Commentscited by 1
  5. 2025Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languagescited by 0