mlsanityData to 7 Oct 2026

LLM360: Towards Fully Transparent Open-Source LLMs

arXiv 2312.06550 ↗First posted 11 Dec 2023cs.CL cs.AI cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present LLM360, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder, including their training code, data, intermediate checkpoints, and analyses (at https://www.llm360.ai). We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

26 of 36 references.

References · 26

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Mistral 7Bcited by 3,065
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
14 more
  1. 2017Mixed Precision Trainingcited by 1,135
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  9. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  11. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  12. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  13. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  14. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 55; 2025: 30; 2026: 14.

100 papers.

Sort:

By area:

  1. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  2. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  5. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  6. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  7. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  8. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  9. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  10. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  11. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  12. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
38 more
  1. 2024Instructional Fingerprinting of Large Language Modelscited by 64
  2. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  3. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  4. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  6. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  7. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  8. 2024IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neuronscited by 36
  9. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 34
  10. 2025Fluid Language Model Benchmarkingcited by 30
  11. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  12. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  13. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 22
  14. 2025Training Optimal Large Diffusion Language Modelscited by 21
  15. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  16. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  17. 2024Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Modelscited by 18
  18. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  19. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  20. 2024UTF:Undertrained Tokens as Fingerprints A Novel Approach to LLM Identificationcited by 16
  21. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  22. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  23. 2024Has this Fact been Edited? Detecting Knowledge Edits in Language Modelscited by 14
  24. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  25. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillationcited by 12
  26. 2024ZipNN: Lossless Compression for AI Modelscited by 12
  27. 2024Risks and Opportunities of Open-Source Generative AIcited by 11
  28. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Modelscited by 10
  29. 2025Independence Tests for Language Modelscited by 10
  30. 2024Do Large Language Models Mirror Cognitive Language Processing?cited by 9
  31. 2024AstroPT: Scaling Large Observation Models for Astronomycited by 9
  32. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 9
  33. 2025MEraser: An Effective Fingerprint Erasure Approach for Large Language Modelscited by 9
  34. 2024ConTrans: Weak-to-Strong Alignment Engineering via Concept Transplantationcited by 8
  35. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 8
  36. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  37. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
  38. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7

50 most cited of 100 All 100 (JSON)