mlsanityData to 7 Oct 2026

Data Science and Technology Towards AGI Part I: Tiered Data Management

arXiv 2602.09003 ↗First posted 9 Feb 2026cs.AI cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

The development of artificial intelligence can be viewed as an evolution of data-driven learning paradigms, with successive shifts in data organization and utilization continuously driving advances in model capability. Current LLM research is dominated by a paradigm that relies heavily on unidirectional scaling of data size, increasingly encountering bottlenecks in data availability, acquisition cost, and training efficiency. In this work, we argue that the development of AGI is entering a new phase of data-model co-evolution, in which models actively guide data management while high-quality data, in turn, amplifies model capabilities. To implement this vision, we propose a tiered data management framework, designed to support the full LLM training lifecycle across heterogeneous learning objectives and cost constraints. Specifically, we introduce an L0-L4 tiered data management framework, ranging from raw uncurated resources to organized and verifiable knowledge. Importantly, LLMs are fully used in data management processes, such as quality scoring and content editing, to refine data across tiers. Each tier is characterized by distinct data properties, management strategies, and training roles, enabling data to be strategically allocated across LLM training stages, including pre-training, mid-training, and alignment. The framework balances data quality, acquisition cost, and marginal training benefit, providing a systematic approach to scalable and sustainable data management. We validate the effectiveness of the proposed framework through empirical studies, in which tiered datasets are constructed from raw corpora and used across multiple training phases. Experimental results demonstrate that tier-aware data utilization significantly improves training efficiency and model performance. To facilitate further research, we release our tiered datasets and processing tools to the community.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 133 of its 161 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

133 of 161 references.

References · 133

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2025Qwen3 Technical Reportcited by 8,195
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  13. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  14. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  15. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  16. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  17. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  18. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  19. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  21. 2021Measuring Coding Challenge Competence With APPScited by 879
  22. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  23. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  24. 2023StarCoder: may the source be with you!cited by 790
  25. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  26. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  27. 2024Yi: Open Foundation Models by 01.AIcited by 706
  28. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  29. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  30. 2023Instruction Tuning with GPT-4cited by 676
  31. 2024Phi-4 Technical Reportcited by 675
  32. 2024Self-Rewarding Language Modelscited by 654
  33. 2023Textbooks Are All You Needcited by 561
  34. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  35. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  36. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  37. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  38. 2025LIMO: Less is More for Reasoningcited by 488

50 most cited of 133 All 133 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.