mlsanityData to 7 Oct 2026

Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models

arXiv 2408.06663 ↗First posted 13 Aug 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

The development of large language models leads to the formation of a pre-train-then-align paradigm, in which the model is typically pre-trained on a large text corpus and undergoes a tuning stage to align the model with human preference or downstream tasks. In this work, we investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints. Our results on 18 datasets suggest that i) continual pre-training improves the model in a latent way that unveils after fine-tuning; ii) with extra fine-tuning, the datasets that the model does not demonstrate capability gain much more than those that the model performs well during the pre-training stage; iii) although model benefits significantly through supervised fine-tuning, it may forget previously known domain knowledge and the tasks that are not seen during fine-tuning; iv) the model resembles high sensitivity to evaluation prompts after supervised fine-tuning, but this sensitivity can be alleviated by more pre-training.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 72 of its 105 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

72 of 105 references.

References · 72

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  13. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  14. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  15. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  16. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  17. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  18. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  19. 2023LIMA: Less Is More for Alignmentcited by 1,080
  20. 2015Teaching Machines to Read and Comprehendcited by 1,039
  21. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  22. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  23. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  24. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  25. 2023Large Language Modelscited by 774
  26. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  27. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  28. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  29. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  30. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  31. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  32. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  33. 2023The Falcon Series of Open Language Modelscited by 526
  34. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  35. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  36. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  37. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  38. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 6.

9 papers.

  1. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  2. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  3. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 14
  4. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 12
  5. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 8
  6. 2026Scaling Participation in Modular AI Systemscited by 2
  7. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  8. 2026The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portabilitycited by 0
  9. 2026Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descentcited by 0