Data Science and Technology Towards AGI Part I: Tiered Data Management
Not on the map. Why?
Abstract (arXiv)
The development of artificial intelligence can be viewed as an evolution of data-driven learning paradigms, with successive shifts in data organization and utilization continuously driving advances in model capability. Current LLM research is dominated by a paradigm that relies heavily on unidirectional scaling of data size, increasingly encountering bottlenecks in data availability, acquisition cost, and training efficiency. In this work, we argue that the development of AGI is entering a new phase of data-model co-evolution, in which models actively guide data management while high-quality data, in turn, amplifies model capabilities. To implement this vision, we propose a tiered data management framework, designed to support the full LLM training lifecycle across heterogeneous learning objectives and cost constraints. Specifically, we introduce an L0-L4 tiered data management framework, ranging from raw uncurated resources to organized and verifiable knowledge. Importantly, LLMs are fully used in data management processes, such as quality scoring and content editing, to refine data across tiers. Each tier is characterized by distinct data properties, management strategies, and training roles, enabling data to be strategically allocated across LLM training stages, including pre-training, mid-training, and alignment. The framework balances data quality, acquisition cost, and marginal training benefit, providing a systematic approach to scalable and sustainable data management. We validate the effectiveness of the proposed framework through empirical studies, in which tiered datasets are constructed from raw corpora and used across multiple training phases. Experimental results demonstrate that tier-aware data utilization significantly improves training efficiency and model performance. To facilitate further research, we release our tiered datasets and processing tools to the community.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 133 of its 161 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
133 of 161 references.
References · 133
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2023Let's Verify Step by Stepcited by 3,598
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2022Competition-Level Code Generation with AlphaCodecited by 1,427
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2016Bag of Tricks for Efficient Text Classificationcited by 869
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
- 2023StarCoder: may the source be with you!cited by 790
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
- 2024Yi: Open Foundation Models by 01.AIcited by 706
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2023Instruction Tuning with GPT-4cited by 676
- 2024Phi-4 Technical Reportcited by 675
- 2024Self-Rewarding Language Modelscited by 654
- 2023Textbooks Are All You Needcited by 561
- 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
- 2024Executable Code Actions Elicit Better LLM Agentscited by 537
- 2025LIMO: Less is More for Reasoningcited by 488
50 most cited of 133 All 133 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.