Training LLM Agents to Empower Humans
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Assistive agents should not only take actions on behalf of a human, but also step out of the way and cede control when there are important decisions to be made. However, current methods for building assistive agents, whether via mimicking expert humans or via RL finetuning on an inferred reward, often encourage agents to complete tasks on their own rather than truly assisting the human attain their objectives. Additionally, these methods often require costly explicit human feedback to provide a training signal. We propose a new approach to tuning assistive language models based on maximizing the human's empowerment, their ability to effect desired changes in the environment. Our empowerment-maximizing method, Empower, only requires offline text data, providing a self-supervised method for fine-tuning language models to better assist humans. To study the efficacy of our approach, we conducted an 18-person user study comparing our empowerment assistant with a strong baseline. Participants preferred our assistant 78% of the time (p=0.015), with a 31% higher acceptance rate and 38% fewer suggestions. Additionally, we introduce a new environment for evaluating multi-turn code assistance using simulated humans. Using this environment, we show that agents trained with Empower increase the success rate of a simulated human programmer on challenging coding questions by an average of 192% over an SFT baseline. With this empowerment objective, we provide a framework for useful aligned AI agents at scale using only offline data without the need for any additional human feedback or verifiable rewards.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 39 of its 71 references that are in the map. First posted on arXiv 15 Oct 2025.
Builds on
39 of 71 references.
Closest ancestors · 1
Same direction, newest first
Other references · 38
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2020Learning to summarize from human feedbackcited by 2,562
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2025Gemma 3 Technical Reportcited by 1,686
26 more
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
- 2022Large Language Models Can Self-Improvecited by 656
- 2024Self-Rewarding Language Modelscited by 654
- 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
- 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
- 2016Cooperative Inverse Reinforcement Learningcited by 298
- 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
- 2017Inverse Reward Designcited by 246
- 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
- 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
- 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
- 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
- 2013Empowerment -- an Introductioncited by 65
- 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
- 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
- 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
- 2020AvE: Assistance via Empowermentcited by 27
- 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
- 2024Learning to Assist Humans without Inferring Rewardscited by 12
- 2025Plasticity as the Mirror of Empowermentcited by 11
- 2018A unified strategy for implementing curiosity and empowerment driven reinforcement learningcited by 9
- 2024A Survey of Calibration Process for Black-Box LLMscited by 8
- 2024LBAP: Improved Uncertainty Alignment of LLM Planners using Bayesian Inferencecited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
- 2026GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents
- 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- 2024Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback
- 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions
- 2024Offline Training of Language Model Agents with Functions as Learnable Weights
- 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs