# Learning Performance Graphs from Demonstrations via Task-Based Evaluations

arXiv: https://arxiv.org/abs/2204.05909 · HTML: https://mlsanity.com/paper/2204.05909/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [Q-Learning for Robust Satisfaction of Signal Temporal Logic Specifications](https://mlsanity.com/paper/1609.07409/) (2016)
- [RTAMT: Online Robustness Monitors from STL](https://mlsanity.com/paper/2005.11827/) (2020)
- [Active Task-Inference-Guided Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2001.09227/) (2020)
- [Learning from Demonstrations using Signal Temporal Logic](https://mlsanity.com/paper/2102.07730/) (2021)
- [Safety-Aware Apprenticeship Learning](https://mlsanity.com/paper/1710.07983/) (2017)
- [Correct-by-synthesis reinforcement learning with temporal logic constraints](https://mlsanity.com/paper/1503.01793/) (2015)
- [Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints](https://mlsanity.com/paper/2005.13109/) (2020)
- [robo-gym -- An Open Source Toolkit for Distributed Deep Reinforcement Learning on Real and Simulated Robots](https://mlsanity.com/paper/2007.02753/) (2020)
- [Event-Based Signal Temporal Logic Synthesis for Single and Multi-Robot Tasks](https://mlsanity.com/paper/2011.00370/) (2020)
- [Formalizing and Guaranteeing* Human-Robot Interaction](https://mlsanity.com/paper/2006.16732/) (2020)
- [Automata Guided Reinforcement Learning With Demonstrations](https://mlsanity.com/paper/1809.06305/) (2018)
- [Study of Signal Temporal Logic Robustness Metrics for Robotic Tasks Optimization](https://mlsanity.com/paper/2110.00339/) (2021)
- [Explaining Reward Functions to Humans for Better Human-Robot Collaboration](https://mlsanity.com/paper/2110.04192/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Safe Preference Learning Approach for Personalization with Applications to Autonomous Vehicles](https://mlsanity.com/paper/2311.02099/) (2023)
- [Signal Temporal Logic-Guided Apprenticeship Learning](https://mlsanity.com/paper/2311.05084/) (2023)
- [Scaling Learning based Policy Optimization for Temporal Logic Tasks by Controller Network Dropout](https://mlsanity.com/paper/2403.15826/) (2024)

## Abstract (arXiv)

In the learning from demonstration (LfD) paradigm, understanding and evaluating the demonstrated behaviors plays a critical role in extracting control policies for robots. Without this knowledge, a robot may infer incorrect reward functions that lead to undesirable or unsafe control policies. Recent work has proposed an LfD framework where a user provides a set of formal task specifications to guide LfD, to address the challenge of reward shaping. However, in this framework, specifications are manually ordered in a performance graph (a partial order that specifies relative importance between the specifications). The main contribution of this paper is an algorithm to learn the performance graph directly from the user-provided demonstrations, and show that the reward functions generated using the learned performance graph generate similar policies to those from manually specified performance graphs. We perform a user study that shows that priorities specified by users on behaviors in a simulated highway driving domain match the automatically inferred performance graph. This establishes that we can accurately evaluate user demonstrations with respect to task specifications without expert criteria.