mlsanityData to 14 Oct 2026

Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning

arXiv 2211.15339 ↗First posted 24 Nov 2022cs.LG cs.AI cs.ROcited by 3

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In this work, we consider one-shot imitation learning for object rearrangement tasks, where an AI agent needs to watch a single expert demonstration and learn to perform the same task in different environments. To achieve a strong generalization, the AI agent must infer the spatial goal specification for the task. However, there can be multiple goal specifications that fit the given demonstration. To address this, we propose a reward learning approach, Graph-based Equivalence Mappings (GEM), that can discover spatial goal representations that are aligned with the intended goal specification, enabling successful generalization in unseen environments. Specifically, GEM represents a spatial goal specification by a reward function conditioned on i) a graph indicating important spatial relationships between objects and ii) state equivalence mappings for each edge in the graph indicating invariant properties of the corresponding relationship. GEM combines inverse reinforcement learning and active reward learning to efficiently improve the reward function by utilizing the graph structure and domain randomization enabled by the equivalence mappings. We conducted experiments with simulated oracles and with human subjects. The results show that GEM can drastically improve the generalizability of the learned goal representations over strong baselines.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 49 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

31 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  10. 2017One-Shot Imitation Learningcited by 233
  11. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  12. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
18 more
  1. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  2. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  3. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  4. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 148
  5. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  6. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  7. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  8. 2019Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoningcited by 61
  9. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  10. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  11. 2017A survey of robot learning from demonstrations for Human-Robot Collaborationcited by 38
  12. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
  13. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 35
  14. 2019Learning One-Shot Imitation from Humans without Humanscited by 33
  15. 2016On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systemscited by 28
  16. 2021My House, My Rules: Learning Tidying Preferences with Graph Neural Networkscited by 27
  17. 2021PHASE: PHysically-grounded Abstract Social Events for Machine Social Perceptioncited by 22
  18. 2021Desk Organization: Effect of Multimodal Inputs on Spatial Relational Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023Benchmarks for Physical Reasoning AIcited by 8
  2. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  3. 2023Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Makingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Learning from Trajectories via Subgoal Discovery
  2. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  3. 2026Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning
  4. 2025Consistent Zero-Shot Imitation with Contrastive Goal Inference
  5. 2025Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning
  6. 2019Learning to Reach Goals via Iterated Supervised Learning
  7. 2026Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
  8. 2022GoalsEye: Learning High Speed Precision Table Tennis on a Physical Robot