mlsanityData to 7 Oct 2026

RedPajama: an Open Dataset for Training Large Language Models

arXiv 2411.12372 ↗First posted 19 Nov 2024cs.CL cs.LGcited by 178

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models are increasingly becoming a cornerstone technology in artificial intelligence, the sciences, and society as a whole, yet the optimal strategies for dataset composition and filtering remain largely elusive. Many of the top-performing models lack transparency in their dataset curation and model development processes, posing an obstacle to the development of fully open language models. In this paper, we identify three core data-related challenges that must be addressed to advance open-source language models. These include (1) transparency in model development, including the data curation process, (2) access to large quantities of high-quality data, and (3) availability of artifacts and metadata for dataset curation and analysis. To address these challenges, we release RedPajama-V1, an open reproduction of the LLaMA training dataset. In addition, we release RedPajama-V2, a massive web-only dataset consisting of raw, unfiltered text data together with quality signals and metadata. Together, the RedPajama datasets comprise over 100 trillion tokens spanning multiple domains and with their quality signals facilitate the filtering of data, aiming to inspire the development of numerous new datasets. To date, these datasets have already been used in the training of strong language models used in production, such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. To provide insight into the quality of RedPajama, we present a series of analyses and ablation studies with decoder-only language models with up to 1.6B parameters. Our findings demonstrate how quality signals for web data can be effectively leveraged to curate high-quality subsets of the dataset, underscoring the potential of RedPajama to advance the development of transparent and high-performing language models at scale.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 30 of its 66 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

30 of 66 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023Mistral 7Bcited by 3,065
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  12. 2023StarCoder: may the source be with you!cited by 790
18 more
  1. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  8. 2023The Falcon Series of Open Language Modelscited by 526
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  12. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  13. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  14. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  15. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  16. 2023The Foundation Model Transparency Indexcited by 63
  17. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  18. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 115; 2026: 57.

178 papers.

Sort:

By area:

  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  2. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  3. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  4. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  5. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 35
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  7. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  8. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  9. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  10. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  11. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  12. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22
38 more
  1. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 22
  2. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  3. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  4. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  5. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  6. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  7. 2025Mid-Training of Large Language Models: A Surveycited by 18
  8. 2025A Survey on LLM Mid-Trainingcited by 18
  9. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Wordscited by 17
  10. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  11. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  12. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 16
  13. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  14. 2025The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguitycited by 15
  15. 2025Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stagescited by 13
  16. 2025HalluEntity: Benchmarking and Understanding Entity-Level Hallucination Detectioncited by 13
  17. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  18. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 12
  19. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  20. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  21. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 11
  22. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  23. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  24. 2026Fast-weight Product Key Memorycited by 11
  25. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  26. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  27. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 10
  28. 2025Measuring and Guiding Monosemanticitycited by 10
  29. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  30. 2025The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generationcited by 9
  31. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  32. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deploymentcited by 9
  33. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  34. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  35. 2025Data Efficacy for Language Model Trainingcited by 8
  36. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levelscited by 8
  37. 2024STAR: Synthesis of Tailored Architecturescited by 7
  38. 2024KV Shifting Attention Enhances Language Modelingcited by 7

50 most cited of 178 All 178 (JSON)