# Understanding reinforcement learned crowds

arXiv: https://arxiv.org/abs/2209.09344 · HTML: https://mlsanity.com/paper/2209.09344/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.10082/) (2017)
- [Deep Reinforcement Learning for Swarm Systems](https://mlsanity.com/paper/1807.06613/) (2018)
- [Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning](https://mlsanity.com/paper/2010.04304/) (2020)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Human-Inspired Multi-Agent Navigation using Knowledge Distillation](https://mlsanity.com/paper/2103.10000/) (2021)
- [A Perceptually-Validated Metric for Crowd Trajectory Quality Evaluation](https://mlsanity.com/paper/2108.12346/) (2021)

## Built on by (most cited first, 2 of 2)

- [UGAE: A Novel Approach to Non-exponential Discounting](https://mlsanity.com/paper/2302.05740/) (2023)
- [Reward Function Design for Crowd Simulation via Reinforcement Learning](https://mlsanity.com/paper/2309.12841/) (2023)

## Abstract (arXiv)

Simulating trajectories of virtual crowds is a commonly encountered task in Computer Graphics. Several recent works have applied Reinforcement Learning methods to animate virtual agents, however they often make different design choices when it comes to the fundamental simulation setup. Each of these choices comes with a reasonable justification for its use, so it is not obvious what is their real impact, and how they affect the results. In this work, we analyze some of these arbitrary choices in terms of their impact on the learning performance, as well as the quality of the resulting simulation measured in terms of the energy efficiency. We perform a theoretical analysis of the properties of the reward function design, and empirically evaluate the impact of using certain observation and action spaces on a variety of scenarios, with the reward function and energy usage as metrics. We show that directly using the neighboring agents' information as observation generally outperforms the more widely used raycasting. Similarly, using nonholonomic controls with egocentric observations tends to produce more efficient behaviors than holonomic controls with absolute observations. Each of these choices has a significant, and potentially nontrivial impact on the results, and so researchers should be mindful about choosing and reporting them in their work.