# Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition

arXiv: https://arxiv.org/abs/1805.11686 · HTML: https://mlsanity.com/paper/1805.11686/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 29 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Built on by (most cited first, 30 of 69)

- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback](https://mlsanity.com/paper/2402.03681/) (2024)
- [SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2401.16013/) (2024)
- [RoboCLIP: One Demonstration is Enough to Learn Robot Policies](https://mlsanity.com/paper/2310.07899/) (2023)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation](https://mlsanity.com/paper/2109.01115/) (2021)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?](https://mlsanity.com/paper/2204.11134/) (2022)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [How to Leverage Unlabeled Data in Offline Reinforcement Learning](https://mlsanity.com/paper/2202.01741/) (2022)
- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning](https://mlsanity.com/paper/2310.15145/) (2023)
- [Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification](https://mlsanity.com/paper/2103.12656/) (2021)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback](https://mlsanity.com/paper/2108.05382/) (2021)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)
- [Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.01488/) (2023)
- [Rank2Reward: Learning Shaped Reward Functions from Passive Video](https://mlsanity.com/paper/2404.14735/) (2024)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination](https://mlsanity.com/paper/2310.05208/) (2023)
- [Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance](https://mlsanity.com/paper/2212.09902/) (2022)
- [Affordance-Guided Reinforcement Learning via Visual Prompting](https://mlsanity.com/paper/2407.10341/) (2024)
- [A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning](https://mlsanity.com/paper/2205.05212/) (2022)
- [Continuously Improving Mobile Manipulation with Autonomous Real-World RL](https://mlsanity.com/paper/2409.20568/) (2024)
- [Privileged Sensing Scaffolds Reinforcement Learning](https://mlsanity.com/paper/2405.14853/) (2024)
- [MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations](https://mlsanity.com/paper/2303.17156/) (2023)
- [Outcome-Driven Reinforcement Learning via Variational Inference](https://mlsanity.com/paper/2104.10190/) (2021)
- [Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning](https://mlsanity.com/paper/2107.03961/) (2021)

## Abstract (arXiv)

The design of a reward function often poses a major practical challenge to real-world applications of reinforcement learning. Approaches such as inverse reinforcement learning attempt to overcome this challenge, but require expert demonstrations, which can be difficult or expensive to obtain in practice. We propose variational inverse control with events (VICE), which generalizes inverse reinforcement learning methods to cases where full demonstrations are not needed, such as when only samples of desired goal states are available. Our method is grounded in an alternative perspective on control and reinforcement learning, where an agent's goal is to maximize the probability that one or more events will happen at some point in the future, rather than maximizing cumulative rewards. We demonstrate the effectiveness of our methods on continuous control tasks, with a focus on high-dimensional observations like images where rewards are hard or even impossible to specify.