mlsanityData to 7 Oct 2026

Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents

arXiv 2606.18537 ↗First posted 16 Jun 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Humans often acquire new skills by observing others, since observed behaviors implicitly reveal how to act reasonably in an environment. However, observations drawn from a heterogeneous population introduce conflicting behavioral signals, making it difficult to determine which behaviors are worth imitating. We address this challenge with General Reward Inference and Disentanglement (GRID), a social learning method that extracts universally useful behaviors from a heterogeneous population of demonstrators pursuing different goals. GRID decomposes per-agent reward functions into a general reward, capturing behaviors shared across all agents, and specific rewards, capturing individual preferences and objectives, through an information bottleneck. Training exclusively on the general reward provides a new paradigm of generalist pretraining. It yields a generalist agent that internalizes universal environmental competencies, such as safety and basic task proficiency, without the mode-averaging bias that afflicts standard learning from demonstration techniques. This generalist serves as a strong prior for fine-tuning to downstream tasks, including preferences unseen during training. Experiments across a synthetic basis function decomposition, multi-agent Craftax, continuous control tasks (MuJoCo Gym) and an autonomous driving simulator (Highway-Env) confirm that GRID successfully disentangles reward structure in a semantically meaningful way, outperforms standard learning from demonstration baselines, and enables more efficient and stable specialization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 49 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

31 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience9 Aug
  2. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables20 Sep

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2016Deep Variational Information Bottleneckcited by 915
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2018Behavioral Cloning from Observationcited by 482
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  12. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
17 more
  1. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  2. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  4. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  5. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  6. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  7. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 57
  8. 2017Multimodal Probabilistic Model-Based Planning for Human-Robot Interactioncited by 50
  9. 2020Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approachcited by 47
  10. 2020Emergent Social Learning via Multi-agent Reinforcement Learningcited by 44
  11. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  12. 2023MAVERIC: A Data-Driven Approach to Personalized Autonomous Drivingcited by 22
  13. 2020Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillationcited by 19
  14. 2022Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrationscited by 14
  15. 2021Successor Feature Representationscited by 6
  16. 2021Improving Robot-Centric Learning from Demonstration via Personalized Embeddingscited by 2
  17. 2026Learning Probabilistic Responsibility Allocations for Multi-Agent Interactionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains
  2. 2021Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data
  3. 2025Reflective Verbal Reward Design for Pluralistic Alignment
  4. 2022Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences
  5. 2025AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
  6. 2026Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization
  7. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptation
  8. 2024Value Internalization: Learning and Generalizing from Social Reward