# SLIM: Skill Learning with Multiple Critics

arXiv: https://arxiv.org/abs/2402.00823 · HTML: https://mlsanity.com/paper/2402.00823/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies](https://mlsanity.com/paper/1905.09808/) (2019)
- [Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2109.13841/) (2021)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning](https://mlsanity.com/paper/2302.11289/) (2023)
- [Controllability-Aware Unsupervised Skill Discovery](https://mlsanity.com/paper/2302.05103/) (2023)
- [Mutual Information State Intrinsic Control](https://mlsanity.com/paper/2103.08107/) (2021)
- [Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery](https://mlsanity.com/paper/2204.13906/) (2022)

## Built on by (most cited first, 3 of 3)

- [Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies](https://mlsanity.com/paper/2605.11387/) (2026)
- [Trajectory First: A Curriculum for Discovering Diverse Policies](https://mlsanity.com/paper/2506.01568/) (2025)
- [Hierarchical Lead Critic based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.21680/) (2026)

## Abstract (arXiv)

Self-supervised skill learning aims to acquire useful behaviors that leverage the underlying dynamics of the environment. Latent variable models, based on mutual information maximization, have been successful in this task but still struggle in the context of robotic manipulation. As it requires impacting a possibly large set of degrees of freedom composing the environment, mutual information maximization fails alone in producing useful and safe manipulation behaviors. Furthermore, tackling this by augmenting skill discovery rewards with additional rewards through a naive combination might fail to produce desired behaviors. To address this limitation, we introduce SLIM, a multi-critic learning approach for skill discovery with a particular focus on robotic manipulation. Our main insight is that utilizing multiple critics in an actor-critic framework to gracefully combine multiple reward functions leads to a significant improvement in latent-variable skill discovery for robotic manipulation while overcoming possible interference occurring among rewards which hinders convergence to useful skills. Furthermore, in the context of tabletop manipulation, we demonstrate the applicability of our novel skill discovery approach to acquire safe and efficient motor primitives in a hierarchical reinforcement learning fashion and leverage them through planning, significantly surpassing baseline approaches for skill discovery.