# Personalized LLM Response Generation with Parameterized Memory Injection

arXiv: https://arxiv.org/abs/2404.03565 · HTML: https://mlsanity.com/paper/2404.03565/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [Revisiting Parameter-Efficient Tuning: Are We Really There Yet?](https://mlsanity.com/paper/2202.07962/) (2022)
- [AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2301.12132/) (2023)
- [Less is More: Learning to Refine Dialogue History for Personalized Dialogue Generation](https://mlsanity.com/paper/2204.08128/) (2022)
- [One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles](https://mlsanity.com/paper/2108.09355/) (2021)
- [Kformer: Knowledge Injection in Transformer Feed-Forward Layers](https://mlsanity.com/paper/2201.05742/) (2022)
- [Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement](https://mlsanity.com/paper/2204.13074/) (2022)
- [Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback](https://mlsanity.com/paper/2112.09737/) (2021)
- [Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories](https://mlsanity.com/paper/2306.05406/) (2023)
- [Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation](https://mlsanity.com/paper/2209.11000/) (2022)

## Built on by (most cited first, 30 of 33)

- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [LLMs + Persona-Plug = Personalized LLMs](https://mlsanity.com/paper/2409.11901/) (2024)
- [How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulation](https://mlsanity.com/paper/2502.14642/) (2025)
- [PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Process](https://mlsanity.com/paper/2507.04607/) (2025)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation](https://mlsanity.com/paper/2409.15240/) (2024)
- [Evaluating Long-Context Reasoning in LLM-Based WebAgents](https://mlsanity.com/paper/2512.04307/) (2025)
- [TeleMem: Building Long-Term and Multimodal Memory for Agentic AI](https://mlsanity.com/paper/2601.06037/) (2025)
- [CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steering](https://mlsanity.com/paper/2507.04756/) (2025)
- [Personalized LLM Decoding via Contrasting Personal Preference](https://mlsanity.com/paper/2506.12109/) (2025)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction](https://mlsanity.com/paper/2511.13410/) (2025)
- [How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants](https://mlsanity.com/paper/2601.16621/) (2026)
- [Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning](https://mlsanity.com/paper/2410.08014/) (2024)
- [Mitigating Memorization in LLMs using Activation Steering](https://mlsanity.com/paper/2503.06040/) (2025)
- [The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs](https://mlsanity.com/paper/2510.09905/) (2025)
- [CURP: Codebook-based Continuous User Representation for Personalized Generation with LLMs](https://mlsanity.com/paper/2602.00742/) (2026)
- [EXACT: Explicit Attribute-Guided Decoding-Time Personalization](https://mlsanity.com/paper/2602.17695/) (2026)
- [CARD: Cluster-level Adaptation with Reward-guided Decoding for Personalized Text Generation](https://mlsanity.com/paper/2601.06352/) (2026)
- [Rehearse With User: Personalized Opinion Summarization via Role-Playing based on Large Language Models](https://mlsanity.com/paper/2503.00449/) (2025)
- [Multi-Objective Neural Architecture Search by Learning Search Space Partitions](https://mlsanity.com/paper/2406.00291/) (2024)
- [TSUBASA: Improving Long-Horizon Personalization via Evolving Memory and Self-Learning with Context Distillation](https://mlsanity.com/paper/2604.07894/) (2026)
- [Beyond Isolated Behaviors: Hierarchical User Modeling for LLM Personalization](https://mlsanity.com/paper/2606.02300/) (2026)
- [VitalDiagnosis: AI-Driven Ecosystem for 24/7 Vital Monitoring and Chronic Disease Management](https://mlsanity.com/paper/2601.15798/) (2026)
- [User-Aware Active Knowledge Acquisition for Emotional Support Dialogue](https://mlsanity.com/paper/2605.29715/) (2026)
- [Training-Free VLM Personalization via Calibrated Residual Decoding](https://mlsanity.com/paper/2608.22263/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have exhibited remarkable proficiency in comprehending and generating natural language. On the other hand, personalized LLM response generation holds the potential to offer substantial benefits for individuals in critical areas such as medical. Existing research has explored memory-augmented methods to prompt the LLM with pre-stored user-specific knowledge for personalized response generation in terms of new queries. We contend that such paradigm is unable to perceive fine-granularity information. In this study, we propose a novel Memory-injected approach using parameter-efficient fine-tuning (PEFT) and along with a Bayesian Optimisation searching strategy to achieve LLM Personalization(MiLP).