# PerPLM: Personalized Fine-tuning of Pretrained Language Models via Writer-specific Intermediate Learning and Prompts

arXiv: https://arxiv.org/abs/2309.07727 · HTML: https://mlsanity.com/paper/2309.07727/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Personalized Prompt Learning for Explainable Recommendation](https://mlsanity.com/paper/2202.07371/) (2022)
- [NPA: Neural News Recommendation with Personalized Attention](https://mlsanity.com/paper/1907.05559/) (2019)
- [Exploiting BERT for End-to-End Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1910.00883/) (2019)
- [UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysis](https://mlsanity.com/paper/2110.00135/) (2021)
- [Personalized Language Model for Query Auto-Completion](https://mlsanity.com/paper/1804.09661/) (2018)
- [NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging](https://mlsanity.com/paper/2112.00405/) (2021)
- [Compositional Demographic Word Embeddings](https://mlsanity.com/paper/2010.02986/) (2020)
- [Dynamic Contextualized Word Embeddings](https://mlsanity.com/paper/2010.12684/) (2020)
- [Abusive Language Detection with Graph Convolutional Networks](https://mlsanity.com/paper/1904.04073/) (2019)
- [mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2110.08151/) (2021)
- [Exploring the Value of Personalized Word Embeddings](https://mlsanity.com/paper/2011.06057/) (2020)
- [You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in NLP](https://mlsanity.com/paper/1909.00412/) (2019)
- [Representing Social Media Users for Sarcasm Detection](https://mlsanity.com/paper/1808.08470/) (2018)
- [Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network](https://mlsanity.com/paper/1701.03578/) (2017)
- [Toward Word Embedding for Personalized Information Retrieval](https://mlsanity.com/paper/1606.06991/) (2016)
- [Examination and Extension of Strategies for Improving Personalized Language Modeling via Interpolation](https://mlsanity.com/paper/2006.05469/) (2020)

## Built on by (most cited first, 3 of 3)

- [Personalized Text Generation with Fine-Grained Linguistic Control](https://mlsanity.com/paper/2402.04914/) (2024)
- [Multi-Objective Linguistic Control of Large Language Models](https://mlsanity.com/paper/2406.16229/) (2024)
- [JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew](https://mlsanity.com/paper/2604.18041/) (2026)

## Abstract (arXiv)

The meanings of words and phrases depend not only on where they are used (contexts) but also on who use them (writers). Pretrained language models (PLMs) are powerful tools for capturing context, but they are typically pretrained and fine-tuned for universal use across different writers. This study aims to improve the accuracy of text understanding tasks by personalizing the fine-tuning of PLMs for specific writers. We focus on a general setting where only the plain text from target writers are available for personalization. To avoid the cost of fine-tuning and storing multiple copies of PLMs for different users, we exhaustively explore using writer-specific prompts to personalize a unified PLM. Since the design and evaluation of these prompts is an underdeveloped area, we introduce and compare different types of prompts that are possible in our setting. To maximize the potential of prompt-based personalized fine-tuning, we propose a personalized intermediate learning based on masked language modeling to extract task-independent traits of writers' text. Our experiments, using multiple tasks, datasets, and PLMs, reveal the nature of different prompts and the effectiveness of our intermediate learning approach.