# Variational Meta Reinforcement Learning for Social Robotics

arXiv: https://arxiv.org/abs/2206.03211 · HTML: https://mlsanity.com/paper/2206.03211/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 87 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Decentralized Non-communicating Multiagent Collision Avoidance with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.07845/) (2016)
- [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1805.01956/) (2018)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning](https://mlsanity.com/paper/1812.00971/) (2018)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Discovering Reinforcement Learning Algorithms](https://mlsanity.com/paper/2007.08794/) (2020)
- [Safe Reinforcement Learning with Model Uncertainty Estimates](https://mlsanity.com/paper/1810.08700/) (2018)
- [Improving Generalization in Meta Reinforcement Learning using Learned Objectives](https://mlsanity.com/paper/1910.04098/) (2019)
- [The Usual Suspects? Reassessing Blame for VAE Posterior Collapse](https://mlsanity.com/paper/1912.10702/) (2019)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Some Considerations on Learning to Explore via Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.01118/) (2018)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [Improving Explorability in Variational Inference with Annealed Variational Objectives](https://mlsanity.com/paper/1809.01818/) (2018)
- [A Survey on Human-aware Robot Navigation](https://mlsanity.com/paper/2106.11650/) (2021)
- [MELD: Meta-Reinforcement Learning from Images via Latent State Models](https://mlsanity.com/paper/2010.13957/) (2020)
- [Multi-agent Motion Planning for Dense and Dynamic Environments via Deep Reinforcement Learning](https://mlsanity.com/paper/2001.06627/) (2020)
- [Reinforcement Learning Approaches in Social Robotics](https://mlsanity.com/paper/2009.09689/) (2020)
- [Deep-RBF Networks Revisited: Robust Classification with Rejection](https://mlsanity.com/paper/1812.03190/) (2018)
- [On Embodied Visual Navigation in Real Environments Through Habitat](https://mlsanity.com/paper/2010.13439/) (2020)
- [Deep RBFNet: Point Cloud Feature Learning using Radial Basis Functions](https://mlsanity.com/paper/1812.04302/) (2018)

## Built on by (most cited first, 3 of 3)

- [Demonstration-Enhanced Adaptable Multi-Objective Robot Navigation](https://mlsanity.com/paper/2404.04857/) (2024)
- [Optimal Gait Control for a Tendon-driven Soft Quadruped Robot by Model-based Reinforcement Learning](https://mlsanity.com/paper/2406.07069/) (2024)
- [Few-shot Scooping Under Domain Shift via Simulated Maximal Deployment Gaps](https://mlsanity.com/paper/2408.02949/) (2024)

## Abstract (arXiv)

With the increasing presence of robots in our every-day environments, improving their social skills is of utmost importance. Nonetheless, social robotics still faces many challenges. One bottleneck is that robotic behaviors need to be often adapted as social norms depend strongly on the environment. For example, a robot should navigate more carefully around patients in a hospital compared to workers in an office. In this work, we investigate meta-reinforcement learning (meta-RL) as a potential solution. Here, robot behaviors are learned via reinforcement learning where a reward function needs to be chosen so that the robot learns an appropriate behavior for a given environment. We propose to use a variational meta-RL procedure that quickly adapts the robots' behavior to new reward functions. As a result, given a new environment different reward functions can be quickly evaluated and an appropriate one selected. The procedure learns a vectorized representation for reward functions and a meta-policy that can be conditioned on such a representation. Given observations from a new reward function, the procedure identifies its representation and conditions the meta-policy to it. While investigating the procedures' capabilities, we realized that it suffers from posterior collapse where only a subset of the dimensions in the representation encode useful information resulting in a reduced performance. Our second contribution, a radial basis function (RBF) layer, partially mitigates this negative effect. The RBF layer lifts the representation to a higher dimensional space, which is more easily exploitable for the meta-policy. We demonstrate the interest of the RBF layer and the usage of meta-RL for social robotics on four robotic simulation tasks.