# Online Inverse Reinforcement Learning via Bellman Gradient Iteration

arXiv: https://arxiv.org/abs/1707.09393 · HTML: https://mlsanity.com/paper/1707.09393/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2017.

## Built on by (most cited first, 1 of 1)

- [Iterative Teacher-Aware Learning](https://mlsanity.com/paper/2110.00137/) (2021)

## Abstract (arXiv)

This paper develops an online inverse reinforcement learning algorithm aimed at efficiently recovering a reward function from ongoing observations of an agent's actions. To reduce the computation time and storage space in reward estimation, this work assumes that each observed action implies a change of the Q-value distribution, and relates the change to the reward function via the gradient of Q-value with respect to reward function parameter. The gradients are computed with a novel Bellman Gradient Iteration method that allows the reward function to be updated whenever a new observation is available. The method's convergence to a local optimum is proved. This work tests the proposed method in two simulated environments, and evaluates the algorithm's performance under a linear reward function and a non-linear reward function. The results show that the proposed algorithm only requires a limited computation time and storage space, but achieves an increasing accuracy as the number of observations grows. We also present a potential application to robot cleaners at home.