# A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings

arXiv: https://arxiv.org/abs/2104.08492 · HTML: https://mlsanity.com/paper/2104.08492/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 15 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [Building Generalizable Agents with a Realistic and Rich 3D Environment](https://mlsanity.com/paper/1801.02209/) (2018)
- [Shaping Belief States with Generative Environment Models for RL](https://mlsanity.com/paper/1906.09237/) (2019)
- [Dynamics-aware Embeddings](https://mlsanity.com/paper/1908.09357/) (2019)
- [Neural SLAM: Learning to Explore with External Memory](https://mlsanity.com/paper/1706.09520/) (2017)
- [Forward Modeling for Partial Observation Strategy Games - A StarCraft Defogger](https://mlsanity.com/paper/1812.00054/) (2018)

## Built on by (most cited first, 1 of 1)

- [GroundControl: Anticipating Navigation Failures in Vision-Language Agents via Trajectory-Consistent Uncertainty Estimates](https://mlsanity.com/paper/2606.20479/) (2026)

## Abstract (arXiv)

In this work we explore an auxiliary loss useful for reinforcement learning in environments where strong performing agents are required to be able to navigate a spatial environment. The auxiliary loss proposed is to minimize the classification error of a neural network classifier that predicts whether or not a pair of states sampled from the agents current episode trajectory are in order. The classifier takes as input a pair of states as well as the agent's memory. The motivation for this auxiliary loss is that there is a strong correlation with which of a pair of states is more recent in the agents episode trajectory and which of the two states is spatially closer to the agent. Our hypothesis is that learning features to answer this question encourages the agent to learn and internalize in memory representations of states that facilitate spatial reasoning. We tested this auxiliary loss on a navigation task in a gridworld and achieved 9.6% increase in accumulative episode reward compared to a strong baseline approach.