# A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Model

arXiv: https://arxiv.org/abs/2304.08109 · HTML: https://mlsanity.com/paper/2304.08109/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections](https://mlsanity.com/paper/2104.04670/) (2021)
- [Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases](https://mlsanity.com/paper/2303.14742/) (2023)
- [ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization](https://mlsanity.com/paper/2201.06910/) (2022)

## Built on by (most cited first, 19 of 19)

- [Learning From Mistakes Makes LLM Better Reasoner](https://mlsanity.com/paper/2310.20689/) (2023)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes](https://mlsanity.com/paper/2312.06353/) (2023)
- [Efficient RLHF: Reducing the Memory Usage of PPO](https://mlsanity.com/paper/2309.00754/) (2023)
- [Aligning Large Language Models with Human: A Survey](https://mlsanity.com/paper/2307.12966/) (2023)
- [Understanding Layer Significance in LLM Alignment](https://mlsanity.com/paper/2410.17875/) (2024)
- [ConPET: Continual Parameter-Efficient Tuning for Large Language Models](https://mlsanity.com/paper/2309.14763/) (2023)
- [Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals](https://mlsanity.com/paper/2305.18425/) (2023)
- [HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy](https://mlsanity.com/paper/2401.15207/) (2024)
- [SambaLingo: Teaching Large Language Models New Languages](https://mlsanity.com/paper/2404.05829/) (2024)
- [The Primacy of Magnitude in Low-Rank Adaptation](https://mlsanity.com/paper/2507.06558/) (2025)
- [Towards Incremental Learning in Large Language Models: A Critical Review](https://mlsanity.com/paper/2404.18311/) (2024)
- [Using fine-tuning and min lookahead beam search to improve Whisper](https://mlsanity.com/paper/2309.10299/) (2023)
- [Learnware of Language Models: Specialized Small Language Models Can Do Big](https://mlsanity.com/paper/2505.13425/) (2025)
- [Token-Efficient Leverage Learning in Large Language Models](https://mlsanity.com/paper/2404.00914/) (2024)
- [Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation](https://mlsanity.com/paper/2504.12436/) (2025)
- [Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysis](https://mlsanity.com/paper/2507.02904/) (2025)
- [Code-Based English Models Surprising Performance on Chinese QA Pair Extraction Task](https://mlsanity.com/paper/2401.10286/) (2024)
- [Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI](https://mlsanity.com/paper/2508.08270/) (2025)

## Abstract (arXiv)

Recently, the instruction-tuning of large language models is a crucial area of research in the field of natural language processing. Due to resource and cost limitations, several researchers have employed parameter-efficient tuning techniques, such as LoRA, for instruction tuning, and have obtained encouraging results In comparison to full-parameter fine-tuning, LoRA-based tuning demonstrates salient benefits in terms of training costs. In this study, we undertook experimental comparisons between full-parameter fine-tuning and LoRA-based tuning methods, utilizing LLaMA as the base model. The experimental results show that the selection of the foundational model, training dataset scale, learnable parameter quantity, and model training cost are all important factors. We hope that the experimental conclusions of this paper can provide inspiration for training large language models, especially in the field of Chinese, and help researchers find a better trade-off strategy between training cost and model performance. To facilitate the reproduction of the paper's results, the dataset, model and code will be released.