Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization
Not on the map. Why?
Abstract (arXiv)
This paper presents a systematic methodology for building domain-specific Japanese small language models using QLoRA fine-tuning. We address three core questions: optimal training scale, base-model selection, and architecture-aware quantization. Stage 1 (Training scale): Scale-learning experiments (1k--5k samples) identify n=4,000 as optimal, where test-set NLL reaches minimum (1.127) before overfitting at 5k samples. Stage 2 (Compare finetuned SLMs): Comparing four Japanese LLMs shows that Llama-3 models with Japanese continual pre-training (Swallow-8B, ELYZA-JP-8B) outperform multilingual models (Qwen2.5-7B). Stage 3 (Quantization): Llama-3 architectures improve under Q4_K_M quantization, while GQA architectures degrade severely (Qwen2.5: -0.280 points). Production recommendation: Swallow-8B Q4_K_M achieves 2.830/3 score, 8.9 s/question, 4.9 GB size. The methodology generalizes to low-resource technical domains and provides actionable guidance for compact Japanese specialist LMs on consumer hardware.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 12 Mar 2026.
Builds on
21 of 44 references.
References · 21
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023A Survey of Large Language Modelscited by 2,537
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
9 more
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
- 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
- 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
- 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
- 2024Release of Pre-Trained Models for the Japanese Languagecited by 22
- 2024Building a Large Japanese Web Corpus for Large Language Modelscited by 11
Built on by
Citing papers per year: 2026: 2.
2 papers.