# Personalizing Large Language Model Agents with Small Policy Models

arXiv: https://arxiv.org/abs/2608.00215 · HTML: https://mlsanity.com/paper/2608.00215/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 28 of 28)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [Conservative Contextual Linear Bandits](https://mlsanity.com/paper/1611.06426/) (2016)
- [Safe Linear Thompson Sampling with Side Information](https://mlsanity.com/paper/1911.02156/) (2019)
- [Contextual Bandits with Large Action Spaces: Made Practical](https://mlsanity.com/paper/2207.05836/) (2022)
- [Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs](https://mlsanity.com/paper/2502.19148/) (2025)
- [IntelligentPooling: Practical Thompson Sampling for mHealth](https://mlsanity.com/paper/2008.01571/) (2020)
- [Action Centered Contextual Bandits](https://mlsanity.com/paper/1711.03596/) (2017)
- [Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedback](https://mlsanity.com/paper/1901.00301/) (2019)
- [Learning Personalized Agents from Human Feedback](https://mlsanity.com/paper/2602.16173/) (2026)
- [PersonaAgent: Bridging Memory and Action for Personalized LLM Agents](https://mlsanity.com/paper/2506.06254/) (2025)
- [Factored Bandits](https://mlsanity.com/paper/1807.01488/) (2018)
- [T-POP: Test-Time Personalization with Online Preference Feedback](https://mlsanity.com/paper/2509.24696/) (2025)
- [Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents](https://mlsanity.com/paper/2607.13591/) (2026)
- [ADAPT: Actively Discovering and Adapting to Preferences for any Task](https://mlsanity.com/paper/2504.04040/) (2025)
- [MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedback](https://mlsanity.com/paper/2606.07909/) (2026)
- [User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction](https://mlsanity.com/paper/2603.20939/) (2026)
- [SAGER: Self-Evolving User Policy Skills for Recommendation Agent](https://mlsanity.com/paper/2604.14972/) (2026)
- [Latent Preference Modeling for Multi-Session Personalized Tool Calling](https://mlsanity.com/paper/2604.17886/) (2026)
- [OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents](https://mlsanity.com/paper/2605.11169/) (2026)

## Abstract (arXiv)

Large language model (LLM) agents can retrieve memory, call tools, ask clarifying questions, and vary response style, yet adapting these execution decisions to an individual user remains difficult. Fine-tuning a separate LLM is costly or impossible for proprietary systems, while prompts and memory primarily expose user information to the agent rather than adapt its execution decisions from feedback. We formulate personalization of a frozen agent as online learning of a per-user execution policy from scalar feedback observed only for the executed action. We propose FABLE (Factorized Adaptive Bandit Layer for Execution), a lightweight policy layer outside a potentially black-box host agent. FABLE factorizes memory, information-acquisition, and response decisions so feedback updates related choices; filters actions through an externally specified feasible set before exploration; and learns user-specific residual preferences relative to a fixed default-and-cost score via Bayesian contextual Thompson sampling. Under a linear residual-reward model, a calibrated variant inherits an expected-regret bound against the best feasible action. We also characterize preferences unidentifiable under persistent feasibility constraints and provide anytime-valid false-promotion control. Across personalized-reasoning, controlled-feedback, and executable tool-use evaluations, FABLE improves several preference-sensitive behaviors relative to rule-only control while remaining competitive on end-to-end task performance.