mlsanityData to 14 Oct 2026

SlimPajama-DC: Understanding Data Combinations for LLM Training

arXiv 2309.10818 ↗First posted 19 Sep 2023cs.CL cs.AIcited by 72

Not on the map. Why?

Abstract (arXiv)

This paper aims to understand the impacts of various data combinations (e.g., web text, Wikipedia, GitHub, books) on the pretraining of large language models using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source dataset, which has been refined and further deduplicated to 627B tokens from the extensive 1.2T token RedPajama dataset contributed by Together. We have termed our research as SlimPajama-DC, an empirical analysis designed to uncover fundamental characteristics and best practices associated with employing SlimPajama in the training of large language models. During our research with SlimPajama, two pivotal observations emerged: (1) Global deduplication vs. local deduplication. We analyze and discuss how global (across different sources of datasets) and local (within the single source of dataset) deduplications affect the performance of trained models. (2) Proportions of highly-deduplicated multi-source datasets in the combination. To study this, we construct six configurations on SlimPajama dataset and train individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best configuration outperforms the 1.3B model trained on RedPajama using the same number of training tokens by a significant margin. All our 1.3B models are trained on Cerebras 16× CS-2 cluster with a total of 80 PFLOP/s in bf16 mixed precision. We further extend our discoveries (such as increasing data diversity is crucial after global deduplication) on a 7B model with large batch-size training. Our SlimPajama-DC models are available at: https://huggingface.co/MBZUAI-LLM/SlimPajama-DC and the separate SlimPajama-DC datasets are available at: https://huggingface.co/datasets/MBZUAI-LLM/SlimPajama-627B-DC.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

28 of 48 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2020GLU Variants Improve Transformercited by 1,804
16 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  8. 2016Language Modeling with Gated Convolutional Networkscited by 702
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  13. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  14. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  15. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  16. 2023Dropout Reduces Underfittingcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 34; 2025: 20; 2026: 14.

72 papers.

Sort:

By area:

  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  2. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  3. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  4. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  5. 2023Calibrated Language Models Must Hallucinatecited by 115
  6. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  7. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  8. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 67
  9. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  10. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  11. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 46
  12. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
38 more
  1. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  2. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  3. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 26
  4. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  5. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 23
  6. 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
  7. 2025Mid-Training of Large Language Models: A Surveycited by 21
  8. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  9. 2023Data Management For Training Large Language Models: A Surveycited by 17
  10. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  11. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  12. 2024γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Modelscited by 14
  13. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  14. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  15. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  16. 2026Reasoning Models Generate Societies of Thoughtcited by 12
  17. 2024From Deception to Detection: The Dual Roles of Large Language Models in Fake Newscited by 11
  18. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  19. 2024House of Cards: Massive Weights in LLMscited by 9
  20. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 8
  21. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 8
  22. 2025HuggingGraph: Understanding the Supply Chain of LLM Ecosystemcited by 8
  23. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  24. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  25. 2023YAYI 2: Multilingual Open-Source Large Language Modelscited by 6
  26. 2024Benchmarking General-Purpose In-Context Learningcited by 6
  27. 2024SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Modelscited by 6
  28. 2025On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspectivecited by 6
  29. 2024DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimizationcited by 5
  30. 2024Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Trainingcited by 5
  31. 20247B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancementcited by 5
  32. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 5
  33. 2025Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-trainingcited by 5
  34. 2025A Survey of LLM × DATAcited by 4
  35. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 4
  36. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  37. 2025Mixtera: A Data Plane for Foundation Model Trainingcited by 3
  38. 2024EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignmentcited by 2

50 most cited of 72 All 72 (JSON)