mlsanityData to 14 Oct 2026

PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

arXiv 2502.01033 ↗First posted 3 Feb 2025cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed underlinePrompt underlineAware underlineRepresentation underlineAdjustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 31 of its 73 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

31 of 73 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
19 more
  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  2. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  3. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  5. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  6. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  7. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  9. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  10. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  11. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 93
  12. 2023PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domaincited by 40
  13. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  14. 2023Model-as-a-Service (MaaS): A Surveycited by 25
  15. 2023F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Taskscited by 18
  16. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 17
  17. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  18. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  19. 2022Continuous Detection, Rapidly React: Unseen Rumors Detection based on Continual Prompt-Tuningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025AMAS: Adaptively Determining Communication Topology for LLM-based Multi-Agent Systemcited by 15
  2. 2025FT-MDT: Extracting Decision Trees from Medical Texts via a Novel Low-rank Adaptation Methodcited by 7
  3. 2026MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicinecited by 4
  4. 2026Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyondcited by 1
  5. 2026Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domaincited by 1
  6. 2026ACL: Aligned Contrastive Learning Improves BERT and Multi-exit BERT Fine-tuningcited by 1
  7. 2025Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptationcited by 0
  8. 2026A Generative Model for Joint Multiple Intent Detection and Slot Fillingcited by 0
  9. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0