# Distilling Realizable Students from Unrealizable Teachers

arXiv: https://arxiv.org/abs/2505.09546 · HTML: https://mlsanity.com/paper/2505.09546/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 14 May 2025.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 21 of 21)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [In-Hand Object Rotation via Rapid Motor Adaptation](https://mlsanity.com/paper/2210.04887/) (2022)
- [Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes](https://mlsanity.com/paper/2211.11744/) (2022)
- [From Imitation to Refinement -- Residual RL for Precise Assembly](https://mlsanity.com/paper/2407.16677/) (2024)
- [Learning Montezuma's Revenge from a Single Demonstration](https://mlsanity.com/paper/1812.03381/) (2018)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [Bridging the Imitation Gap by Adaptive Insubordination](https://mlsanity.com/paper/2007.12173/) (2020)
- [Sequence Model Imitation Learning with Unobserved Contexts](https://mlsanity.com/paper/2208.02225/) (2022)
- [SPIN: Simultaneous Perception, Interaction and Navigation](https://mlsanity.com/paper/2405.07991/) (2024)
- [Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation](https://mlsanity.com/paper/2303.03486/) (2023)
- [Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback](https://mlsanity.com/paper/2410.05434/) (2024)
- [Robust Asymmetric Learning in POMDPs](https://mlsanity.com/paper/2012.15566/) (2020)
- [Privileged Sensing Scaffolds Reinforcement Learning](https://mlsanity.com/paper/2405.14853/) (2024)
- [Student-Informed Teacher Training](https://mlsanity.com/paper/2412.09149/) (2024)

## Built on by (most cited first, 2 of 2)

- [End-to-end RL Improves Dexterous Grasping Policies](https://mlsanity.com/paper/2509.16434/) (2025)
- [Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient](https://mlsanity.com/paper/2605.26478/) (2026)

## Abstract (arXiv)

We study policy distillation under privileged information, where a student policy with only partial observations must learn from a teacher with full-state access. A key challenge is information asymmetry: the student cannot directly access the teacher's state space, leading to distributional shifts and policy degradation. Existing approaches either modify the teacher to produce realizable but sub-optimal demonstrations or rely on the student to explore missing information independently, both of which are inefficient. Our key insight is that the student should strategically interact with the teacher --querying only when necessary and resetting from recovery states --to stay on a recoverable path within its own observation space. We introduce two methods: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration. We validate our methods in both simulated and real-world robotic tasks, demonstrating significant improvements over standard teacher-student baselines in training efficiency and final performance. The project website is available at : https://portal-cornell.github.io/CritiQ_ReTRy/