# Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles

arXiv: https://arxiv.org/abs/2503.16978 · HTML: https://mlsanity.com/paper/2503.16978/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Models](https://mlsanity.com/paper/2406.04806/) (2024)
- [Counter-Strike Deathmatch with Large-Scale Behavioural Cloning](https://mlsanity.com/paper/2104.04258/) (2021)
- [Boosting Continuous Control with Consistency Policy](https://mlsanity.com/paper/2310.06343/) (2023)
- [Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.04080/) (2024)
- [Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning](https://mlsanity.com/paper/2208.07811/) (2022)
- [Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Games](https://mlsanity.com/paper/2307.11105/) (2023)
- [Efficient Active Imitation Learning with Random Network Distillation](https://mlsanity.com/paper/2411.01894/) (2024)
- [Generating Personas for Games with Multimodal Adversarial Imitation Learning](https://mlsanity.com/paper/2308.07598/) (2023)

## Built on by (most cited first, 3 of 3)

- [Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps](https://mlsanity.com/paper/2505.10482/) (2025)
- [TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2506.22008/) (2025)
- [A Hierarchical Approach to Imitation Learning for Manipulation Tasks Requiring Time Varying Forces](https://mlsanity.com/paper/2608.03103/) (2026)

## Abstract (arXiv)

Diffusion models have shown impressive performance in capturing complex and multi-modal action distributions for game agents, but their slow inference speed prevents practical deployment in real-time game environments. While consistency models offer a promising approach for one-step generation, they often suffer from training instability and performance degradation when applied to policy learning. In this paper, we present CPQE (Consistency Policy with Q-Ensembles), which combines consistency models with Q-ensembles to address these challenges.CPQE leverages uncertainty estimation through Q-ensembles to provide more reliable value function approximations, resulting in better training stability and improved performance compared to classic double Q-network methods. Our extensive experiments across multiple game scenarios demonstrate that CPQE achieves inference speeds of up to 60 Hz -- a significant improvement over state-of-the-art diffusion policies that operate at only 20 Hz -- while maintaining comparable performance to multi-step diffusion approaches. CPQE consistently outperforms state-of-the-art consistency model approaches, showing both higher rewards and enhanced training stability throughout the learning process. These results indicate that CPQE offers a practical solution for deploying diffusion-based policies in games and other real-time applications where both multi-modal behavior modeling and rapid inference are critical requirements.