mlsanityData to 14 Oct 2026

Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach

arXiv 2510.20235 ↗First posted 23 Oct 2025cs.LG cs.AIcited by 5

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

In this paper, we propose a provably convergent and practical framework for multi-objective reinforcement learning with max-min criterion. From a game-theoretic perspective, we reformulate max-min multi-objective reinforcement learning as a two-player zero-sum regularized continuous game and introduce an efficient algorithm based on mirror descent. Our approach simplifies the policy update while ensuring global last-iterate convergence. We provide a comprehensive theoretical analysis on our algorithm, including iteration complexity under both exact and approximate policy evaluations, as well as sample complexity bounds. To further enhance performance, we modify the proposed algorithm with adaptive regularization. Our experiments demonstrate the convergence behavior of the proposed algorithm in tabular settings, and our implementation for deep reinforcement learning significantly outperforms previous baselines in many MORL environments.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 65 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

37 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Intersectional Fairness in Reinforcement Learning with Large State and Constraint Spaces17 Feb
  2. 2024The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm12 Jun

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2021Convex optimizationcited by 2,515
  4. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  5. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  7. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  8. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  9. 2020Exploration-Exploitation in Constrained MDPscited by 149
  10. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  11. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 109
  12. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 101
23 more
  1. 2021Last-iterate Convergence of Decentralized Optimistic Gradient Descent/Ascent in Infinite-horizon Competitive Markov Gamescited by 69
  2. 2019Reinforcement Learning with Convex Constraintscited by 60
  3. 2020Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewardscited by 60
  4. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 58
  5. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  6. 2019Distributionally Robust Reinforcement Learningcited by 42
  7. 2020Distributional Robustness and Regularization in Reinforcement Learningcited by 40
  8. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 40
  9. 2018Urban Driving with Multi-Objective Deep Reinforcement Learningcited by 30
  10. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  11. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
  12. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  13. 2022Welfare and Fairness in Multi-objective Reinforcement Learningcited by 20
  14. 2022Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Gamescited by 18
  15. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 16
  16. 2022Efficient Phi-Regret Minimization in Extensive-Form Games via Online Mirror Descentcited by 15
  17. 2022Alternating Mirror Descent for Constrained Min-Max Gamescited by 13
  18. 2022Decentralized Optimistic Hyperpolicy Mirror Descent: Provably No-Regret Learning in Markov Gamescited by 12
  19. 2023Can We Find Nash Equilibria at a Linear Rate in Markov Games?cited by 9
  20. 2023Adaptively Perturbed Mirror Descent for Learning in Gamescited by 9
  21. 2023Fairness in Preference-based Reinforcement Learningcited by 9
  22. 2021A Deep Reinforcement Learning Approach for Fair Traffic Signal Controlcited by 7
  23. 2024Policy Optimization finds Nash Equilibrium in Regularized General-Sum LQ Gamescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  2. 2026Multi-Objective Reinforcement Learning for Large-Scale Tote Allocation in Human-Robot Collaborative Fulfillment Centerscited by 0
  3. 2026Auction-Based Online Policy Adaptation for Evolving Objectivescited by 0
  4. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Modelscited by 0
  5. 2026Constrained Multi-Objective Reinforcement Learning with Max-Min Criterioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
  2. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
  3. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
  4. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games
  5. 2024Toward Finding Strong Pareto Optimal Policies in Multi-Agent Reinforcement Learning
  6. 2026Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization
  7. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  8. 2026Maximum Likelihood Reinforcement Learning