# Playing FPS Games with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1609.05521 · HTML: https://mlsanity.com/paper/1609.05521/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2016.

## Built on by (most cited first, 30 of 95)

- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Temporal Predictive Coding For Model-Based Planning In Latent Space](https://mlsanity.com/paper/2106.07156/) (2021)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Counter-Strike Deathmatch with Large-Scale Behavioural Cloning](https://mlsanity.com/paper/2104.04258/) (2021)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Augmenting Automated Game Testing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.15819/) (2021)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2206.10442/) (2022)
- [Chain of Thought Imitation with Procedure Cloning](https://mlsanity.com/paper/2205.10816/) (2022)
- [SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction](https://mlsanity.com/paper/2503.18211/) (2025)
- [Recurrent Off-policy Baselines for Memory-based Continuous Control](https://mlsanity.com/paper/2110.12628/) (2021)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Teaching Agents how to Map: Spatial Reasoning for Multi-Object Navigation](https://mlsanity.com/paper/2107.06011/) (2021)
- [MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines](https://mlsanity.com/paper/2603.06679/) (2026)
- [Branching Time Active Inference: the theory and its generality](https://mlsanity.com/paper/2111.11107/) (2021)
- [Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractions](https://mlsanity.com/paper/2302.12003/) (2023)
- [A Novel Deep Reinforcement Learning Based Automated Stock Trading System Using Cascaded LSTM Networks](https://mlsanity.com/paper/2212.02721/) (2022)
- [Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness](https://mlsanity.com/paper/2309.16973/) (2023)
- [Robotic Surgery With Lean Reinforcement Learning](https://mlsanity.com/paper/2105.01006/) (2021)
- [Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomes](https://mlsanity.com/paper/2207.13649/) (2022)
- [Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games](https://mlsanity.com/paper/2303.02160/) (2023)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [Phy-Q as a measure for physical reasoning intelligence](https://mlsanity.com/paper/2108.13696/) (2021)
- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [Branching Time Active Inference: empirical study and complexity class analysis](https://mlsanity.com/paper/2111.11276/) (2021)
- [Online Learning of Reusable Abstract Models for Object Goal Navigation](https://mlsanity.com/paper/2203.02583/) (2022)
- [Towards Source-free Domain Adaptive Semantic Segmentation via Importance-aware and Prototype-contrast Learning](https://mlsanity.com/paper/2306.01598/) (2023)
- [A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Games](https://mlsanity.com/paper/2509.03682/) (2025)

## Abstract (arXiv)

Advances in deep reinforcement learning have allowed autonomous agents to perform well on Atari games, often outperforming humans, using only raw pixels to make their decisions. However, most of these games take place in 2D environments that are fully observable to the agent. In this paper, we present the first architecture to tackle 3D environments in first-person shooter games, that involve partially observable states. Typically, deep reinforcement learning methods only utilize visual input for training. We present a method to augment these models to exploit game feature information such as the presence of enemies or items, during the training phase. Our model is trained to simultaneously learn these features along with minimizing a Q-learning objective, which is shown to dramatically improve the training speed and performance of our agent. Our architecture is also modularized to allow different models to be independently trained for different phases of the game. We show that the proposed architecture substantially outperforms built-in AI agents of the game as well as humans in deathmatch scenarios.