# A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2607.14373 · HTML: https://mlsanity.com/paper/2607.14373/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Deep Hedging](https://mlsanity.com/paper/1802.03042/) (2018)
- [Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2004.10888/) (2020)
- [Reinforcement Learning with Dynamic Convex Risk Measures](https://mlsanity.com/paper/2112.13414/) (2021)
- [Risk-Aware High-level Decisions for Automated Driving at Occluded Intersections with Reinforcement Learning](https://mlsanity.com/paper/2004.04450/) (2020)
- [Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning](https://mlsanity.com/paper/2102.03119/) (2021)
- [RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes](https://mlsanity.com/paper/2405.04714/) (2024)
- [Robo-advising: Learning Investors' Risk Preferences via Portfolio Choices](https://mlsanity.com/paper/1911.02067/) (2019)
- [Interactive Inverse Reinforcement Learning for Cooperative Games](https://mlsanity.com/paper/2111.04698/) (2021)
- [Personalized Robo-Advising: Enhancing Investment through Client Interaction](https://mlsanity.com/paper/1911.01391/) (2019)
- [Eliciting Risk Aversion with Inverse Reinforcement Learning via Interactive Questioning](https://mlsanity.com/paper/2308.08427/) (2023)
- [Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions](https://mlsanity.com/paper/2505.04553/) (2025)

## Abstract (arXiv)

We propose a noise-robust elicit-to-optimize framework that integrates inverse reinforcement learning (IRL) and reinforcement learning (RL) for eliciting agents' risk preferences and optimizing policies under a broad class of risk objectives characterized by distortion riskmetrics. On the elicitation side, we propose an adaptive Bayesian IRL method that infers agents' latent risk objectives from their noisy observed decisions, explicitly allowing agents to take stochastic and suboptimal actions. We establish the existence of a finite set of distinguishing questions that identifies the preferred distortion riskmetric within the candidate class and prove that the convergence rate of the algorithm is of order O(exp(-cm+O(√mlog m))) under general settings, where c>0 is a constant and m denotes the number of algorithm iterations. On the optimization side, we develop a model-free RL algorithm for optimizing policies under conditional distortion riskmetrics. By representing the objective as an integral of the conditional cost quantile function with respect to the distortion function, the method unifies distortion-riskmetric objectives. We optimize diverse risk objectives by extending the Proximal Policy Optimization (PPO) algorithm with policy, value, and quantile neural networks, where the quantile network estimates the full conditional cost quantile function and enables numerical evaluation of general risk objectives. A comprehensive empirical study demonstrates the framework's elicitation accuracy and effectiveness in complex financial environments.