# Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

arXiv: https://arxiv.org/abs/2501.13126 · HTML: https://mlsanity.com/paper/2501.13126/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2025.

## Built on by (most cited first, 16 of 16)

- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning](https://mlsanity.com/paper/2506.11300/) (2025)
- [A Survey on LLM Mid-Training](https://mlsanity.com/paper/2510.23081/) (2025)
- [Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation](https://mlsanity.com/paper/2505.08364/) (2025)
- [CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2602.03048/) (2026)
- [Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learning](https://mlsanity.com/paper/2512.10534/) (2025)
- [DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models](https://mlsanity.com/paper/2603.26164/) (2026)
- [FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy](https://mlsanity.com/paper/2502.05551/) (2025)
- [An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift](https://mlsanity.com/paper/2601.05882/) (2026)
- [Progressive Multi-Agent Reasoning for Biological Perturbation Prediction](https://mlsanity.com/paper/2602.07408/) (2026)
- [FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training](https://mlsanity.com/paper/2502.00761/) (2025)
- [HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model](https://mlsanity.com/paper/2605.11255/) (2026)
- [Estimating the Effects of Sample Training Orders for Large Language Models without Retraining](https://mlsanity.com/paper/2505.22042/) (2025)
- [REAR: Test-time Preference Realignment through Reward Decomposition](https://mlsanity.com/paper/2606.30339/) (2026)
- [Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling](https://mlsanity.com/paper/2609.32484/) (2026)
- [What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use](https://mlsanity.com/paper/2609.32991/) (2026)

## Abstract (arXiv)

Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that its data preferences dynamically change, indicating the need for pretraining with different data at various training stages. To achieve it, we propose the Perplexity Difference (PD) based Preference Curriculum learning (PDPC) framework, which always perceives and uses the data preferred by LLMs to train and boost them. First, we introduce the PD metric to quantify the difference in how challenging a sample is for weak versus strong models. Samples with high PD are more challenging for weak models to learn and are more suitable to be arranged in the later stage of pretraining. Second, we propose the preference function to approximate and predict the data preference of the LLM at any training step, so as to complete the arrangement of the dataset offline and ensure continuous training without interruption. Experimental results on 1.3B and 3B models demonstrate that PDPC significantly surpasses baselines. Notably, the 3B model trained on 1T tokens achieves an increased average accuracy of over 8.1% across MMLU and CMMLU.