# Generalizing Skills with Semi-Supervised Reinforcement Learning

arXiv: https://arxiv.org/abs/1612.00429 · HTML: https://mlsanity.com/paper/1612.00429/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2016.

## Built on by (most cited first, 18 of 18)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents](https://mlsanity.com/paper/2107.05686/) (2021)
- [Lifelong Inverse Reinforcement Learning](https://mlsanity.com/paper/2207.00461/) (2022)
- [TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning](https://mlsanity.com/paper/2512.13106/) (2025)
- [Robust Subtask Learning for Compositional Generalization](https://mlsanity.com/paper/2302.02984/) (2023)
- [Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment](https://mlsanity.com/paper/2410.23680/) (2024)
- [Sample-Optimal Agnostic Boosting with Unlabeled Data](https://mlsanity.com/paper/2503.04706/) (2025)
- [MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer](https://mlsanity.com/paper/2506.01623/) (2025)
- [Reinforcement Learning for Freight Booking Control Problems](https://mlsanity.com/paper/2102.00092/) (2021)
- [A new soft computing method for integration of expert's knowledge in reinforcement learn-ing problems](https://mlsanity.com/paper/2106.07088/) (2021)
- [The Sandbox Environment for Generalizable Agent Research (SEGAR)](https://mlsanity.com/paper/2203.10351/) (2022)
- [PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward](https://mlsanity.com/paper/2306.01731/) (2023)
- [Zero-Shot Transfer in Imitation Learning](https://mlsanity.com/paper/2310.06710/) (2023)
- [Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Random](https://mlsanity.com/paper/2606.20206/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) can acquire complex behaviors from low-level inputs, such as images. However, real-world applications of such methods require generalizing to the vast variability of the real world. Deep networks are known to achieve remarkable generalization when provided with massive amounts of labeled data, but can we provide this breadth of experience to an RL agent, such as a robot? The robot might continuously learn as it explores the world around it, even while deployed. However, this learning requires access to a reward function, which is often hard to measure in real-world domains, where the reward could depend on, for example, unknown positions of objects or the emotional state of the user. Conversely, it is often quite practical to provide the agent with reward functions in a limited set of situations, such as when a human supervisor is present or in a controlled setting. Can we make use of this limited supervision, and still benefit from the breadth of experience an agent might collect on its own? In this paper, we formalize this problem as semisupervised reinforcement learning, where the reward function can only be evaluated in a set of "labeled" MDPs, and the agent must generalize its behavior to the wide range of states it might encounter in a set of "unlabeled" MDPs, by using experience from both settings. Our proposed method infers the task objective in the unlabeled MDPs through an algorithm that resembles inverse RL, using the agent's own prior experience in the labeled MDPs as a kind of demonstration of optimal behavior. We evaluate our method on challenging tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available. We also show that our method outperforms direct supervised learning of the reward.