# Reward Function Design for Crowd Simulation via Reinforcement Learning

arXiv: https://arxiv.org/abs/2309.12841 · HTML: https://mlsanity.com/paper/2309.12841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.10082/) (2017)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Discounted Reinforcement Learning Is Not an Optimization Problem](https://mlsanity.com/paper/1910.02140/) (2019)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Human-Inspired Multi-Agent Navigation using Knowledge Distillation](https://mlsanity.com/paper/2103.10000/) (2021)
- [UGAE: A Novel Approach to Non-exponential Discounting](https://mlsanity.com/paper/2302.05740/) (2023)
- [Understanding reinforcement learned crowds](https://mlsanity.com/paper/2209.09344/) (2022)
- [Emotional Contagion-Aware Deep Reinforcement Learning for Antagonistic Crowd Simulation](https://mlsanity.com/paper/2105.00854/) (2021)

## Abstract (arXiv)

Crowd simulation is important for video-games design, since it enables to populate virtual worlds with autonomous avatars that navigate in a human-like manner. Reinforcement learning has shown great potential in simulating virtual crowds, but the design of the reward function is critical to achieving effective and efficient results. In this work, we explore the design of reward functions for reinforcement learning-based crowd simulation. We provide theoretical insights on the validity of certain reward functions according to their analytical properties, and evaluate them empirically using a range of scenarios, using the energy efficiency as the metric. Our experiments show that directly minimizing the energy usage is a viable strategy as long as it is paired with an appropriately scaled guiding potential, and enable us to study the impact of the different reward components on the behavior of the simulated crowd. Our findings can inform the development of new crowd simulation techniques, and contribute to the wider study of human-like navigation.