mlsanityData to 14 Oct 2026

Thresholded Lexicographic Ordered Multiobjective Reinforcement Learning

arXiv 2408.13493 ↗First posted 24 Aug 2024cs.LG cs.AIcited by 8

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Lexicographic multi-objective problems, which impose a lexicographic importance order over the objectives, arise in many real-life scenarios. Existing Reinforcement Learning work directly addressing lexicographic tasks has been scarce. The few proposed approaches were all noted to be heuristics without theoretical guarantees as the Bellman equation is not applicable to them. Additionally, the practical applicability of these prior approaches also suffers from various issues such as not being able to reach the goal state. While some of these issues have been known before, in this work we investigate further shortcomings, and propose fixes for improving practical performance in many cases. We also present a policy optimization approach using our Lexicographic Projection Optimization (LPO) algorithm that has the potential to address these theoretical and practical concerns. Finally, we demonstrate our proposed algorithms on benchmark problems.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 36 references that are in the map. First posted on arXiv 24 Aug 2024.

Builds on

12 of 36 references.

References · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  4. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  5. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  6. 2019Pareto Multi-Task Learningcited by 224
  7. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  8. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  9. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  10. 2015Safety-Constrained Reinforcement Learning for MDPscited by 33
  11. 2018Urban Driving with Multi-Objective Deep Reinforcement Learningcited by 30
  12. 2022Lexicographic Multi-Objective Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2026MoralityGym: A Benchmark for Evaluating Hierarchical Moral Alignment in Sequential Decision-Making Agentscited by 2
  2. 2026Efficient Online Lexicographic Generalized Low-Rank Matrix Banditscited by 2
  3. 2025Concealed Adversarial attacks on neural networks for sequential datacited by 1
  4. 2025LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Explorationcited by 1
  5. 2025ProSocialAlign: Preference Conditioned Test Time Alignment in Language Modelscited by 1
  6. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchycited by 0
  7. 2026Reinforcement Learning with a Bilevel World-Model Architecture for Scan-Order Optimisation in Laser Directed Energy Depositioncited by 0
  8. 2026Lexicographic Multi-Objective On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspective
  2. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
  3. 2024The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm
  4. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  5. 2025Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach
  6. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
  7. 2024MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization
  8. 2025Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations