# PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

arXiv: https://arxiv.org/abs/2502.01033 · HTML: https://mlsanity.com/paper/2502.01033/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 73 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [Unified Demonstration Retriever for In-Context Learning](https://mlsanity.com/paper/2305.04320/) (2023)
- [Masking as an Efficient Alternative to Finetuning for Pretrained Language Models](https://mlsanity.com/paper/2004.12406/) (2020)
- [PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain](https://mlsanity.com/paper/2310.14151/) (2023)
- [A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation](https://mlsanity.com/paper/2203.01670/) (2022)
- [Model-as-a-Service (MaaS): A Survey](https://mlsanity.com/paper/2311.05804/) (2023)
- [F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks](https://mlsanity.com/paper/2305.11916/) (2023)
- [Sparse Structure Search for Parameter-Efficient Tuning](https://mlsanity.com/paper/2206.07382/) (2022)
- [Overview of the PromptCBLUE Shared Task in CHIP2023](https://mlsanity.com/paper/2312.17522/) (2023)
- [Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation](https://mlsanity.com/paper/2301.11503/) (2023)

## Built on by (most cited first, 9 of 9)

- [AMAS: Adaptively Determining Communication Topology for LLM-based Multi-Agent System](https://mlsanity.com/paper/2510.01617/) (2025)
- [FT-MDT: Extracting Decision Trees from Medical Texts via a Novel Low-rank Adaptation Method](https://mlsanity.com/paper/2510.04655/) (2025)
- [MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine](https://mlsanity.com/paper/2601.16503/) (2026)
- [Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond](https://mlsanity.com/paper/2601.21767/) (2026)
- [Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domain](https://mlsanity.com/paper/2602.03368/) (2026)
- [ACL: Aligned Contrastive Learning Improves BERT and Multi-exit BERT Fine-tuning](https://mlsanity.com/paper/2602.03563/) (2026)
- [Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation](https://mlsanity.com/paper/2510.24619/) (2025)
- [A Generative Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2602.08322/) (2026)
- [Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations](https://mlsanity.com/paper/2610.00928/) (2026)

## Abstract (arXiv)

In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed underlinePrompt underlineAware underlineRepresentation underlineAdjustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.