Learning to Steer Markovian Agents under Model Uncertainty
Not on the map. Why?
Abstract (arXiv)
Designing incentives for an adapting population is a ubiquitous problem in a wide array of economic applications and beyond. In this work, we study how to design additional rewards to steer multi-agent systems towards desired policies without prior knowledge of the agents' underlying learning dynamics. Motivated by the limitation of existing works, we consider a new and general category of learning dynamics called Markovian agents. We introduce a model-based non-episodic Reinforcement Learning (RL) formulation for our steering problem. Importantly, we focus on learning a history-dependent steering strategy to handle the inherent model uncertainty about the agents' learning dynamics. We introduce a novel objective function to encode the desiderata of achieving a good steering outcome with reasonable cost. Theoretically, we identify conditions for the existence of steering strategies to guide agents to the desired policies. Complementing our theoretical contributions, we provide empirical algorithms to approximately solve our objective, which effectively tackles the challenge in learning history-dependent strategies. We demonstrate the efficacy of our algorithms through empirical evaluations.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 80 references that are in the map. First posted on arXiv 14 Jul 2024.
Builds on
45 of 80 references.
References · 45
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Exploration by Random Network Distillationcited by 966
- 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
- 2016Deep Exploration via Bootstrapped DQNcited by 518
- 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
- 2018Is Q-learning Provably Efficient?cited by 485
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
- 2017Learning with Opponent-Learning Awarenesscited by 218
- 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
- 2017Cycles in adversarial regularized learningcited by 122
- 2022On the Convergence Rates of Policy Gradient Methodscited by 115
- 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
33 more
- 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
- 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
- 2023A Definition of Continual Reinforcement Learningcited by 98
- 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
- 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
- 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
- 2021Near-Optimal No-Regret Learning in General Gamescited by 85
- 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
- 2020Learning to Incentivize Other Learning Agentscited by 55
- 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
- 2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 53
- 2019Strategizing against No-regret Learnerscited by 48
- 2022Model-Free Opponent Shapingcited by 43
- 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
- 2019Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Gamescited by 38
- 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
- 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
- 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 24
- 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
- 2020End-to-End Learning and Intervention in Gamescited by 19
- 2021Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learningcited by 18
- 2022On the convergence of policy gradient methods to Nash equilibria in general stochastic gamescited by 13
- 2023On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximationcited by 13
- 2024Near-Optimal Policy Optimization for Correlated Equilibrium in General-Sum Markov Gamescited by 11
- 2021Coordinating Followers to Reach Better Equilibria: End-to-End Gradient Descent for Stackelberg Gamescited by 10
- 2018Adaptive Mechanism Design: Learning to Promote Cooperationcited by 9
- 2024Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RLcited by 9
- 2021Inducing Equilibria via Incentives: Simultaneous Design-and-Play Ensures Global Convergencecited by 8
- 2024Automated Design of Affine Maximizer Mechanisms in Dynamic Settingscited by 7
- 2018Optimal control policies for evolutionary dynamics with environmental feedbackcited by 2
- 2021In Congestion Games, Taxes Achieve Optimal Approximationcited by 2
- 2022Limit Cycles Analysis and Control of Evolutionary Game Dynamics with Environmental Feedbackcited by 2
- 2023Differentiable Arbitrating in Zero-sum Markov Gamescited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.