mlsanityData to 14 Oct 2026

FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy

arXiv 2502.05551 ↗First posted 8 Feb 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have significantly advanced human language understanding and generation, with pretraining data quality and organization being crucial to their performance. Multi-stage pretraining is a promising approach, but existing methods often lack quantitative criteria for data partitioning and instead rely on intuitive heuristics. In this paper, we propose the novel Four-quadRAnt Multi-stage prEtraining strategy (FRAME), guided by the established principle of organizing the pretraining process into four stages to achieve significant loss reductions four times. This principle is grounded in two key findings: first, training on high Perplexity (PPL) data followed by low PPL data, and second, training on low PPL difference (PD) data followed by high PD data, both causing the loss to drop significantly twice and performance enhancements. By partitioning data into four quadrants and strategically organizing them, FRAME achieves a remarkable 16.8% average improvement over random across MMLU and CMMLU for the 3B model, effectively boosting LLM performance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 8 Feb 2025.

Builds on

37 of 58 references.

References · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
25 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  3. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  7. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  8. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  9. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  10. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  11. 2020The Pushshift Reddit Datasetcited by 292
  12. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  13. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  14. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  15. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  16. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  17. 2024How to Train Data-Efficient LLMscited by 93
  18. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  19. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  20. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  21. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  22. 2021Curriculum learning for language modelingcited by 32
  23. 2025Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Datacited by 16
  24. 2021Multi-stage Pre-training over Simplified Multimodal Pre-training Modelscited by 6
  25. 2025Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrievalcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025A Survey on LLM Mid-Trainingcited by 21
  2. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 12
  3. 2025Estimating the Effects of Sample Training Orders for Large Language Models without Retrainingcited by 0
  4. 2025Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-trainingcited by 0