# Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RL

arXiv: https://arxiv.org/abs/2102.11893 · HTML: https://mlsanity.com/paper/2102.11893/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [Soft Weight-Sharing for Neural Network Compression](https://mlsanity.com/paper/1702.04008/) (2017)
- [Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control](https://mlsanity.com/paper/1708.04133/) (2017)
- [A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning](https://mlsanity.com/paper/1806.07937/) (2018)
- [D2RL: Deep Dense Architectures in Reinforcement Learning](https://mlsanity.com/paper/2010.09163/) (2020)
- [Deep Reinforcement Learning for Navigation in AAA Video Games](https://mlsanity.com/paper/2011.04764/) (2020)
- [Automated Video Game Testing Using Synthetic and Human-Like Agents](https://mlsanity.com/paper/1906.00317/) (2019)
- [Winning Isn't Everything: Enhancing Game Development with Intelligent Agents](https://mlsanity.com/paper/1903.10545/) (2019)
- [On Multi-Agent Learning in Team Sports Games](https://mlsanity.com/paper/1906.10124/) (2019)

## Built on by (most cited first, 3 of 3)

- [Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning](https://mlsanity.com/paper/2510.01656/) (2025)
- [Optimistic critics can empower small actors](https://mlsanity.com/paper/2506.01016/) (2025)
- [Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning](https://mlsanity.com/paper/2506.04398/) (2025)

## Abstract (arXiv)

Actors and critics in actor-critic reinforcement learning algorithms are functionally separate, yet they often use the same network architectures. This case study explores the performance impact of network sizes when considering actor and critic architectures independently. By relaxing the assumption of architectural symmetry, it is often possible for smaller actors to achieve comparable policy performance to their symmetric counterparts. Our experiments show up to 99% reduction in the number of network weights with an average reduction of 77% over multiple actor-critic algorithms on 9 independent tasks. Given that reducing actor complexity results in a direct reduction of run-time inference cost, we believe configurations of actors and critics are aspects of actor-critic design that deserve to be considered independently, particularly in resource-constrained applications or when deploying multiple actors simultaneously.