mlsanityData to 7 Oct 2026

Aligning Instruction Tuning with Pre-training

arXiv 2501.09368 ↗First posted 16 Jan 2025cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Instruction tuning enhances large language models (LLMs) to follow human instructions across diverse tasks, relying on high-quality datasets to guide behavior. However, these datasets, whether manually curated or synthetically generated, are often narrowly focused and misaligned with the broad distributions captured during pre-training, limiting LLM generalization and effective use of pre-trained knowledge. We propose Aligning Instruction Tuning with Pre-training (AITP), a method that bridges this gap by identifying coverage shortfalls in instruction-tuning datasets and rewriting underrepresented pre-training data into high-quality instruction-response pairs. This approach enriches dataset diversity while preserving task-specific objectives. Evaluations on three fully open LLMs across eight benchmarks demonstrate consistent performance improvements with AITP. Ablations highlight the benefits of adaptive data selection, controlled rewriting, and balanced integration, emphasizing the importance of aligning instruction tuning with pre-training distributions to unlock the full potential of LLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 53 of its 63 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

53 of 63 references.

References · 53

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2024Mixtral of Expertscited by 1,785
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2023Instruction Tuning with GPT-4cited by 676
  11. 2022Large Language Models Can Self-Improvecited by 656
  12. 2024Self-Rewarding Language Modelscited by 654
  13. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  14. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  15. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  16. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  17. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  18. 2023Self-Consuming Generative Models Go MADcited by 239
  19. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  20. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  21. 2023Self-Alignment with Instruction Backtranslationcited by 149
  22. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  23. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  24. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  25. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  26. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  27. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  28. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 62
  29. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 62
  30. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  31. 2022Training Language Models with Language Feedbackcited by 48
  32. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  33. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  34. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  35. 2024Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sourcescited by 27
  36. 2023SELF: Self-Evolution with Language Feedbackcited by 24
  37. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  38. 2024Importance Weighting Can Help Large Language Models Self-Improvecited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 26
  2. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  3. 2025Visual Instruction Bottleneck Tuningcited by 5
  4. 2025Exploring Representation Invariance in Finetuningcited by 1
  5. 2026MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuningcited by 1
  6. 2025Backbone Augmented Training for Adaptationscited by 0
  7. 2025Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Setcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  2. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  3. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning
  4. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
  5. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models
  6. 2024MAmmoTH2: Scaling Instructions from the Web
  7. 2024CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions
  8. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning