# Hybrid Reinforcement Learning with Expert State Sequences

arXiv: https://arxiv.org/abs/1903.04110 · HTML: https://mlsanity.com/paper/1903.04110/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 11 Mar 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 14 of 14)

- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning](https://mlsanity.com/paper/2303.00599/) (2023)
- [Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/2201.09736/) (2022)
- [A Survey of Imitation Learning Methods, Environments and Metrics](https://mlsanity.com/paper/2404.19456/) (2024)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization](https://mlsanity.com/paper/2203.02214/) (2022)
- [Improving Policy Learning via Language Dynamics Distillation](https://mlsanity.com/paper/2210.00066/) (2022)
- [Mimicking Better by Matching the Approximate Action Distribution](https://mlsanity.com/paper/2306.09805/) (2023)
- [D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioning](https://mlsanity.com/paper/2210.14428/) (2022)
- [Learning from Online Videos at Inference Time for Computer-Use Agents](https://mlsanity.com/paper/2511.04137/) (2025)
- [Shaped Policy Search for Evolutionary Strategies using Waypoints](https://mlsanity.com/paper/2105.14639/) (2021)
- [Group-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.05135/) (2022)
- [GAN-MPC: Training Model Predictive Controllers with Parameterized Cost Functions using Demonstrations from Non-identical Experts](https://mlsanity.com/paper/2305.19111/) (2023)

## Abstract (arXiv)

Existing imitation learning approaches often require that the complete demonstration data, including sequences of actions and states, are available. In this paper, we consider a more realistic and difficult scenario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are unobserved. We propose a novel tensor-based model to infer the unobserved actions of the expert state sequences. The policy of the agent is then optimized via a hybrid objective combining reinforcement learning and imitation learning. We evaluated our hybrid approach on an illustrative domain and Atari games. The empirical results show that (1) the agents are able to leverage state expert sequences to learn faster than pure reinforcement learning baselines, (2) our tensor-based action inference model is advantageous compared to standard deep neural networks in inferring expert actions, and (3) the hybrid policy optimization objective is robust against noise in expert state sequences.