# Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment

arXiv: https://arxiv.org/abs/2406.01103 · HTML: https://mlsanity.com/paper/2406.01103/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 59 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 30 of 34)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles](https://mlsanity.com/paper/2001.11231/) (2020)
- [Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning](https://mlsanity.com/paper/1812.00971/) (2018)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Learning to Act by Predicting the Future](https://mlsanity.com/paper/1611.01779/) (2016)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [Creating Pro-Level AI for a Real-Time Fighting Game Using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.03821/) (2019)
- [Normalization Enhances Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2306.00656/) (2023)
- [DareFightingICE Competition: A Fighting Game Sound Design and AI Competition](https://mlsanity.com/paper/2203.01556/) (2022)
- [Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2311.01075/) (2023)
- [Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learning](https://mlsanity.com/paper/2309.01352/) (2023)
- [Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2205.14557/) (2022)
- [Conceptual Reinforcement Learning for Language-Conditioned Tasks](https://mlsanity.com/paper/2303.05069/) (2023)
- [DouZero+: Improving DouDizhu AI by Opponent Modeling and Coach-guided Learning](https://mlsanity.com/paper/2204.02558/) (2022)
- [Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equation](https://mlsanity.com/paper/2404.12754/) (2024)
- [Object-Category Aware Reinforcement Learning](https://mlsanity.com/paper/2210.07802/) (2022)

## Built on by (most cited first, 8 of 8)

- [AVA: Attentive VLM Agent for Mastering StarCraft II](https://mlsanity.com/paper/2503.05383/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement](https://mlsanity.com/paper/2609.01481/) (2026)
- [EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-making](https://mlsanity.com/paper/2508.09586/) (2025)
- [Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach](https://mlsanity.com/paper/2510.23216/) (2025)
- [Improving Sample Efficiency in Multi-Agent Reinforcement Learning for Simulated Football Games via Exploration](https://mlsanity.com/paper/2503.13077/) (2025)
- [RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agents](https://mlsanity.com/paper/2609.34712/) (2026)
- [Enhancing Player Enjoyment with a Two-Tier DRL and LLM-Based Agent System for Fighting Games](https://mlsanity.com/paper/2504.07425/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) agents have demonstrated impressive success in a wide range of game genres. However, existing research primarily focuses on optimizing DRL competence rather than addressing the challenge of prolonged player interaction. In this paper, we propose a practical DRL agent system for fighting games named Sh=ukai, which has been successfully deployed to Naruto Mobile, a popular fighting game with over 100 million registered users. Sh=ukai quantifies the state to enhance generalizability, introducing Heterogeneous League Training (HELT) to achieve balanced competence, generalizability, and training efficiency. Furthermore, Sh=ukai implements specific rewards to align the agent's behavior with human expectations. Sh=ukai's ability to generalize is demonstrated by its consistent competence across all characters, even though it was trained on only 13% of them. Additionally, HELT exhibits a remarkable 22% improvement in sample efficiency. Sh=ukai serves as a valuable training partner for players in Naruto Mobile, enabling them to enhance their abilities and skills.