# Towards Human-Like RL: Taming Non-Naturalistic Behavior in Deep RL via Adaptive Behavioral Costs in 3D Games

arXiv: https://arxiv.org/abs/2309.15484 · HTML: https://mlsanity.com/paper/2309.15484/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 27 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Extreme Q-Learning: MaxEnt RL without Entropy](https://mlsanity.com/paper/2301.02328/) (2023)
- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1901.00569/) (2019)
- [Reinforcement learning with human advice: a survey](https://mlsanity.com/paper/2005.11016/) (2020)
- [Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation](https://mlsanity.com/paper/2105.09637/) (2021)
- [Automated Play-Testing Through RL Based Human-Like Play-Styles Generation](https://mlsanity.com/paper/2211.17188/) (2022)
- [Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games](https://mlsanity.com/paper/2303.02160/) (2023)
- [A Rubric for Human-like Agents and NeuroAI](https://mlsanity.com/paper/2212.04401/) (2022)
- [Leveraging human knowledge in tabular reinforcement learning: A study of human subjects](https://mlsanity.com/paper/1805.05769/) (2018)
- [Deep Reinforcement Learning for Human-Like Driving Policies in Collision Avoidance Tasks of Self-Driving Cars](https://mlsanity.com/paper/2006.04218/) (2020)

## Built on by (most cited first, 4 of 4)

- [Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization](https://mlsanity.com/paper/2511.15055/) (2025)
- [UBCL: A Reinforcement Learning Framework for Controllable and Diverse Player Behaviors](https://mlsanity.com/paper/2512.10835/) (2025)
- [Advantage-based Temporal Attack in Reinforcement Learning](https://mlsanity.com/paper/2602.19582/) (2026)
- [Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization](https://mlsanity.com/paper/2605.30928/) (2026)

## Abstract (arXiv)

In this paper, we propose a new approach called Adaptive Behavioral Costs in Reinforcement Learning (ABC-RL) for training a human-like agent with competitive strength. While deep reinforcement learning agents have recently achieved superhuman performance in various video games, some of these unconstrained agents may exhibit actions, such as shaking and spinning, that are not typically observed in human behavior, resulting in peculiar gameplay experiences. To behave like humans and retain similar performance, ABC-RL augments behavioral limitations as cost signals in reinforcement learning with dynamically adjusted weights. Unlike traditional constrained policy optimization, we propose a new formulation that minimizes the behavioral costs subject to a constraint of the value function. By leveraging the augmented Lagrangian, our approach is an approximation of the Lagrangian adjustment, which handles the trade-off between the performance and the human-like behavior. Through experiments conducted on 3D games in DMLab-30 and Unity ML-Agents Toolkit, we demonstrate that ABC-RL achieves the same performance level while significantly reducing instances of shaking and spinning. These findings underscore the effectiveness of our proposed approach in promoting more natural and human-like behavior during gameplay.