mlsanityData to 7 Oct 2026

Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling

arXiv 2305.08285 ↗First posted 15 May 2023cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

The increasing size of language models raises great research interests in parameter-efficient fine-tuning such as LoRA that freezes the pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note and two public medical dialogue datasets. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation qualities on free-text sequence-to-sequence tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

50 of 68 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2016Sequence-Level Knowledge Distillationcited by 761
  11. 2021GPT Understands, Toocited by 715
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  13. 2017Learning multiple visual domains with residual adapterscited by 553
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  16. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  17. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  18. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  19. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  20. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  21. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  22. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  23. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  24. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  25. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  26. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  27. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  28. 2021Learned Token Pruning for Transformerscited by 138
  29. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  30. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  31. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  32. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  33. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  34. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  35. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  36. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  37. 2022Structured Pruning Adapterscited by 12
  38. 2023Task-oriented Memory-efficient Pruning-Adaptercited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  3. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 4