# LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

arXiv: https://arxiv.org/abs/2407.03963 · HTML: https://mlsanity.com/paper/2407.03963/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 59 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [KoBBQ: Korean Bias Benchmark for Question Answering](https://mlsanity.com/paper/2307.16778/) (2023)
- [Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca](https://mlsanity.com/paper/2309.08958/) (2023)
- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models](https://mlsanity.com/paper/2306.16244/) (2023)

## Built on by (most cited first, 30 of 33)

- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?](https://mlsanity.com/paper/2408.10811/) (2024)
- [Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization](https://mlsanity.com/paper/2502.19261/) (2025)
- [Repetition Neurons: How Do Language Models Produce Repetitions?](https://mlsanity.com/paper/2410.13497/) (2024)
- [The Emergence of Abstract Thought in Large Language Models Beyond Any Language](https://mlsanity.com/paper/2506.09890/) (2025)
- [Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality](https://mlsanity.com/paper/2506.14681/) (2025)
- [Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domain](https://mlsanity.com/paper/2412.20309/) (2024)
- [JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models](https://mlsanity.com/paper/2409.13317/) (2024)
- [Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models](https://mlsanity.com/paper/2409.01584/) (2024)
- [Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model](https://mlsanity.com/paper/2410.22736/) (2024)
- [How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders](https://mlsanity.com/paper/2503.06394/) (2025)
- [Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs](https://mlsanity.com/paper/2412.14471/) (2024)
- [Evaluating Multimodal Large Language Models on Vertically Written Japanese Text](https://mlsanity.com/paper/2511.15059/) (2025)
- [JETHICS: Japanese Ethics Understanding Evaluation Dataset](https://mlsanity.com/paper/2506.16187/) (2025)
- [Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales](https://mlsanity.com/paper/2506.03619/) (2025)
- [BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning](https://mlsanity.com/paper/2506.06955/) (2025)
- [Solar Open Technical Report](https://mlsanity.com/paper/2601.07022/) (2026)
- [XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity](https://mlsanity.com/paper/2605.05662/) (2026)
- [Refined and Segmented Price Sentiment Indices from Survey Comments](https://mlsanity.com/paper/2411.09937/) (2024)
- [Analyzing the Safety of Japanese Large Language Models in Stereotype-Triggering Prompts](https://mlsanity.com/paper/2503.01947/) (2025)
- [Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models](https://mlsanity.com/paper/2503.23714/) (2025)
- [Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models](https://mlsanity.com/paper/2504.09979/) (2025)
- [Do LLMs Need to Think in One Language? Correlation between Latent Language and Task Performance](https://mlsanity.com/paper/2505.21458/) (2025)
- [Cost of Reasoning in non-English Languages: A Case Study on Japanese](https://mlsanity.com/paper/2607.10114/) (2026)
- [Enhancing Japanese Large Language Models with Reasoning Vectors](https://mlsanity.com/paper/2508.02913/) (2025)
- [Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptation](https://mlsanity.com/paper/2510.12115/) (2025)
- [Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality](https://mlsanity.com/paper/2602.09444/) (2026)
- [Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models](https://mlsanity.com/paper/2604.02048/) (2026)
- [Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling](https://mlsanity.com/paper/2604.28075/) (2026)
- [Toward LLMs Beyond English-Centric Development](https://mlsanity.com/paper/2605.15613/) (2026)

## Abstract (arXiv)

This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit https://llm-jp.nii.ac.jp/en/.