mlsanityData to 7 Oct 2026

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

arXiv 2402.17193 ↗First posted 27 Feb 2024cs.CL cs.LGcited by 201

Not on the map. Why?

Cited by 31 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). Builds on 49 of its 57 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

49 of 57 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
37 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  12. 2023PaLM 2 Technical Reportcited by 1,154
  13. 2015Teaching Machines to Read and Comprehendcited by 1,039
  14. 2021Scaling Vision Transformerscited by 993
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  16. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  17. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  18. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  19. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  20. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  21. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  22. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  23. 2023Scaling Data-Constrained Language Modelscited by 391
  24. 2022UL2: Unifying Language Learning Paradigmscited by 292
  25. 2021Scaling Laws for Transfercited by 265
  26. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  27. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  28. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  29. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  30. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  31. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  32. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  33. 2021Scaling Laws for Neural Machine Translationcited by 110
  34. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  35. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  36. 2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
  37. 2022Examining Scaling and Transfer of Language Model Architectures for Machine Translationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 60; 2025: 88; 2026: 53.

201 papers.

Sort:

By area:

  1. 2024LoRA Learns Less and Forgets Lesscited by 313
  2. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  3. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  5. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
  6. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  7. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  8. 2025Scaling Laws for Optimal Data Mixturescited by 49
  9. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  10. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  11. 2024A Scalable Communication Protocol for Networks of Large Language Modelscited by 43
  12. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 41
38 more
  1. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  2. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  3. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  4. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  5. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  6. 2024Scaling Optimal LR Across Token Horizonscited by 35
  7. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  8. 2026When Single-Agent with Skills Replace Multi-Agent Systems and When They Failcited by 34
  9. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  10. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  11. 2025Parallel Scaling Law for Language Modelscited by 29
  12. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  13. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  14. 2024Improving Data Efficiency via Curating LLM-Driven Rating Systemscited by 26
  15. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  16. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  17. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  18. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  19. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  20. 2024OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?cited by 22
  21. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 20
  22. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  23. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruningcited by 19
  24. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  25. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoningcited by 19
  26. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 19
  27. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  28. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  29. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  30. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 16
  31. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15
  32. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 15
  33. 2025How Much is Enough? The Diminishing Returns of Tokenization Training Datacited by 15
  34. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  35. 2025Refine Knowledge of Large Language Models via Adaptive Contrastive Learningcited by 14
  36. 2025LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selectioncited by 14
  37. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotationscited by 14
  38. 2025EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corporacited by 14

50 most cited of 201 All 201 (JSON)