# LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch

arXiv: https://arxiv.org/abs/2501.07124 · HTML: https://mlsanity.com/paper/2501.07124/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2025.

## Built on by (most cited first, 15 of 15)

- [Self-Questioning Language Models](https://mlsanity.com/paper/2508.03682/) (2025)
- [Fluid Language Model Benchmarking](https://mlsanity.com/paper/2509.11106/) (2025)
- [Overcoming the Communication-Performance Tradeoff in LLM Pretraining](https://mlsanity.com/paper/2508.15706/) (2025)
- [Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition](https://mlsanity.com/paper/2505.22251/) (2025)
- [Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality](https://mlsanity.com/paper/2503.07879/) (2025)
- [Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet](https://mlsanity.com/paper/2603.08163/) (2026)
- [KnowRL: Teaching Language Models to Know What They Know](https://mlsanity.com/paper/2510.11407/) (2025)
- [MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources](https://mlsanity.com/paper/2509.25531/) (2025)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)
- [CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion](https://mlsanity.com/paper/2607.05046/) (2026)
- [VLA Foundry: A Unified Framework for Training Vision-Language-Action Models](https://mlsanity.com/paper/2604.19728/) (2026)
- [Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset](https://mlsanity.com/paper/2510.01219/) (2025)
- [Selecting Language Models for Social Science: Start Small, Start Open, and Validate](https://mlsanity.com/paper/2601.10926/) (2026)
- ["Rebuilding" Statistics in the Age of AI: A Town Hall Discussion on Culture, Infrastructure, and Training](https://mlsanity.com/paper/2601.17510/) (2026)
- [Beyond Independent Labels: Schwartz-Geometry Decoding for Human Value Detection](https://mlsanity.com/paper/2607.05052/) (2026)

## Abstract (arXiv)

We detail the training of the LLM360 K2-65B model, scaling up our 360-degree OPEN SOURCE approach to the largest and most powerful models under project LLM360. While open-source LLMs continue to advance, the answer to "How are the largest LLMs trained?" remains unclear within the community. The implementation details for such high-capacity models are often protected due to business considerations associated with their high cost. This lack of transparency prevents LLM researchers from leveraging valuable insights from prior experience, e.g., "What are the best practices for addressing loss spikes?" The LLM360 K2 project addresses this gap by providing full transparency and access to resources accumulated during the training of LLMs at the largest scale. This report highlights key elements of the K2 project, including our first model, K2 DIAMOND, a 65 billion-parameter LLM that surpasses LLaMA-65B and rivals LLaMA2-70B, while requiring fewer FLOPs and tokens. We detail the implementation steps and present a longitudinal analysis of K2 DIAMOND's capabilities throughout its training process. We also outline ongoing projects such as TXT360, setting the stage for future models in the series. By offering previously unavailable resources, the K2 project also resonates with the 360-degree OPEN SOURCE principles of transparency, reproducibility, and accessibility, which we believe are vital in the era of resource-intensive AI research.