mlsanityData to 14 Oct 2026

Adaptive Teaching in Heterogeneous Agents: Balancing Surprise in Sparse Reward Scenarios

arXiv 2405.14199 ↗First posted 23 May 2024cs.RO cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Robot learning from demonstration

Abstract (arXiv)

Learning from Demonstration (LfD) can be an efficient way to train systems with analogous agents by enabling ``Student'' agents to learn from the demonstrations of the most experienced ``Teacher'' agent, instead of training their policy in parallel. However, when there are discrepancies in agent capabilities, such as divergent actuator power or joint angle constraints, naively replicating demonstrations that are out of bounds for the Student's capability can limit efficient learning. We present a Teacher-Student learning framework specifically tailored to address the challenge of heterogeneity between the Teacher and Student agents. Our framework is based on the concept of ``surprise'', inspired by its application in exploration incentivization in sparse-reward environments. Surprise is repurposed to enable the Teacher to detect and adapt to differences between itself and the Student. By focusing on maximizing its surprise in response to the environment while concurrently minimizing the Student's surprise in response to the demonstrations, the Teacher agent can effectively tailor its demonstrations to the Student's specific capabilities and constraints. We validate our method by demonstrating improvements in the Student's learning in control tasks within sparse-reward environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 28 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

10 of 28 references.

References · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  4. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  5. 2019State Alignment-based Imitation Learningcited by 73
  6. 2021Curiosity-Driven Exploration via Latent Bayesian Surprisecited by 22
  7. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  8. 2019Teaching on a Budget in Multi-Agent Deep Reinforcement Learningcited by 14
  9. 2021Learning Feasibility to Imitate Demonstrators with Different Dynamicscited by 10
  10. 2020RAT iLQR: A Risk Auto-Tuning Controller to Optimally Account for Stochastic Model Mismatchcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22

Similar, not cited

Close in wording, no citation either way

  1. 2019Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
  2. 2026Learning from Demonstrations via Capability-Aware Goal Sampling
  3. 2026Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
  4. 2019Interactive Teaching Algorithms for Inverse Reinforcement Learning
  5. 2021Curriculum Design for Teaching via Demonstrations: Theory and Applications
  6. 2019Reinforced Imitation in Heterogeneous Action Space
  7. 2026Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
  8. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrations