# Game State Learning via Game Scene Augmentation

arXiv: https://arxiv.org/abs/2207.01289 · HTML: https://mlsanity.com/paper/2207.01289/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 13 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 11 of 11)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [Unsupervised State Representation Learning in Atari](https://mlsanity.com/paper/1906.08226/) (2019)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [Go-Blend behavior and affect](https://mlsanity.com/paper/2109.13388/) (2021)
- [Learning Task-Independent Game State Representations from Unlabeled Images](https://mlsanity.com/paper/2206.06490/) (2022)
- [Estimates for the Branching Factors of Atari Games](https://mlsanity.com/paper/2107.02385/) (2021)

## Built on by (most cited first, 2 of 2)

- [Towards General Game Representations: Decomposing Games Pixels into Content and Style](https://mlsanity.com/paper/2307.11141/) (2023)
- [Future Research Avenues for Artificial Intelligence in Digital Gaming: An Exploratory Report](https://mlsanity.com/paper/2412.14085/) (2024)

## Abstract (arXiv)

Having access to accurate game state information is of utmost importance for any artificial intelligence task including game-playing, testing, player modeling, and procedural content generation. Self-Supervised Learning (SSL) techniques have shown to be capable of inferring accurate game state information from the high-dimensional pixel input of game footage into compressed latent representations. Contrastive Learning is a popular SSL paradigm where the visual understanding of the game's images comes from contrasting dissimilar and similar game states defined by simple image augmentation methods. In this study, we introduce a new game scene augmentation technique -- named GameCLR -- that takes advantage of the game-engine to define and synthesize specific, highly-controlled renderings of different game states, thereby, boosting contrastive learning performance. We test our GameCLR technique on images of the CARLA driving simulator environment and compare it against the popular SimCLR baseline SSL method. Our results suggest that GameCLR can infer the game's state information from game footage more accurately compared to the baseline. Our proposed approach allows us to conduct game artificial intelligence research by directly utilizing screen pixels as input.