mlsanityData to 14 Oct 2026

Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priors

arXiv 2508.07400 ↗First posted 10 Aug 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

In this paper, we consider the problem of recovering time-varying reward functions from either optimal policies or demonstrations coming from a max entropy reinforcement learning problem. This problem is highly ill-posed without additional assumptions on the underlying rewards. However, in many applications, the rewards are indeed parsimonious, and some prior information is available. We consider two such priors on the rewards: 1) rewards are mostly constant and they change infrequently, 2) rewards can be represented by a linear combination of a small number of feature functions. We first show that the reward identification problem with the former prior can be recast as a sparsification problem subject to linear constraints. Moreover, we give a polynomial-time algorithm that solves this sparsification problem exactly. Then, we show that identifying rewards representable with the minimum number of features can be recast as a rank minimization problem subject to linear constraints, for which convex relaxations of rank can be invoked. In both cases, these observations lead to efficient optimization-based reward identification algorithms. Several examples are given to demonstrate the accuracy of the recovered rewards as well as their generalizability.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 40 references that are in the map. First posted on arXiv 10 Aug 2025.

Builds on

11 of 40 references.

References · 11

  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  3. 2019A Theory of Regularized Markov Decision Processescited by 286
  4. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  5. 2021Identifiability in inverse reinforcement learningcited by 55
  6. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  7. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  8. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
  9. 2020Active Task-Inference-Guided Deep Inverse Reinforcement Learningcited by 17
  10. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  11. 2023Time Dependent Inverse Optimal Control using Trigonometric Basis Functionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Scalable Multi-Task Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  2. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  3. 2026Fast Rates for Inverse Reinforcement Learning
  4. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measures
  5. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  6. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures
  7. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressions
  8. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm