PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency
Not on the map. Why?
Abstract (arXiv)
We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 10 Oct 2024.
Builds on
26 of 36 references.
References · 26
- 2017Attention Is All You Needcited by 59,932
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
14 more
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
- 20218-bit Optimizers via Block-wise Quantizationcited by 395
- 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
- 2020Query-Key Normalization for Transformerscited by 266
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
- 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
- 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
- 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
- 2024Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Wordscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025PLaMo 2 Technical Report
- 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs
- 2026Cost of Reasoning in non-English Languages: A Case Study on Japanese
- 2024Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese
- 2024Compass: Large Multilingual Language Model for South-east Asia
- 2021CPM-2: Large-scale Cost-effective Pre-trained Language Models
- 2024Evolutionary Optimization of Model Merging Recipes
- 2026CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation