mlsanityData to 14 Oct 2026

A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Model

arXiv 2304.08109 ↗First posted 17 Apr 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Recently, the instruction-tuning of large language models is a crucial area of research in the field of natural language processing. Due to resource and cost limitations, several researchers have employed parameter-efficient tuning techniques, such as LoRA, for instruction tuning, and have obtained encouraging results In comparison to full-parameter fine-tuning, LoRA-based tuning demonstrates salient benefits in terms of training costs. In this study, we undertook experimental comparisons between full-parameter fine-tuning and LoRA-based tuning methods, utilizing LLaMA as the base model. The experimental results show that the selection of the foundational model, training dataset scale, learnable parameter quantity, and model training cost are all important factors. We hope that the experimental conclusions of this paper can provide inspiration for training large language models, especially in the field of Chinese, and help researchers find a better trade-off strategy between training cost and model performance. To facilitate the reproduction of the paper's results, the dataset, model and code will be released.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 23 of its 48 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

23 of 48 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
11 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2021GPT Understands, Toocited by 716
  3. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  4. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  5. 2023Pretraining Language Models with Human Preferencescited by 247
  6. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  7. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  8. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  9. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  10. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 86
  11. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 6; 2025: 5.

19 papers.

Sort:

By area:

  1. 2023Learning From Mistakes Makes LLM Better Reasonercited by 101
  2. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  3. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  4. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  5. 2023Aligning Large Language Models with Human: A Surveycited by 32
  6. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  7. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 17
  8. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12
  9. 2024HiFT: A Hierarchical Full Parameter Fine-Tuning Strategycited by 12
  10. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  11. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  12. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
7 more
  1. 2023Using fine-tuning and min lookahead beam search to improve Whispercited by 3
  2. 2025Learnware of Language Models: Specialized Small Language Models Can Do Bigcited by 3
  3. 2024Token-Efficient Leverage Learning in Large Language Modelscited by 2
  4. 2025Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptationcited by 1
  5. 2025Enhancing Sports Strategy with Video Analytics and Data Mining: Assessing the effectiveness of Multimodal LLMs in tennis video analysiscited by 1
  6. 2024Code-Based English Models Surprising Performance on Chinese QA Pair Extraction Taskcited by 0
  7. 2025Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AIcited by 0