# ViSA: Visited-State Augmentation for Generalized Goal-Space Contrastive Reinforcement Learning

arXiv: https://arxiv.org/abs/2603.14887 · HTML: https://mlsanity.com/paper/2603.14887/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 30 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 28 of 28)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [C-Learning: Learning to Achieve Goals via Recursive Classification](https://mlsanity.com/paper/2011.08909/) (2020)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots](https://mlsanity.com/paper/1909.11639/) (2019)
- [A Contrastive Learning Approach for Training Variational Autoencoder Priors](https://mlsanity.com/paper/2010.02917/) (2020)
- [Soft Contrastive Learning for Time Series](https://mlsanity.com/paper/2312.16424/) (2023)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [Contrastive Difference Predictive Coding](https://mlsanity.com/paper/2310.20141/) (2023)
- [Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning](https://mlsanity.com/paper/2207.09081/) (2022)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [MHER: Model-based Hindsight Experience Replay](https://mlsanity.com/paper/2107.00306/) (2021)
- [Nonprehensile Planar Manipulation through Reinforcement Learning with Multimodal Categorical Exploration](https://mlsanity.com/paper/2308.02459/) (2023)
- [Goal-Conditioned Reinforcement Learning with Disentanglement-based Reachability Planning](https://mlsanity.com/paper/2307.10846/) (2023)
- [Unsupervised domain adaptation by learning using privileged information](https://mlsanity.com/paper/2303.09350/) (2023)
- [Goal Exploration Augmentation via Pre-trained Skills for Sparse-Reward Long-Horizon Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2210.16058/) (2022)

## Abstract (arXiv)

Goal-Conditioned Reinforcement Learning (GCRL) is a framework for learning a policy that can reach arbitrarily given goals. In particular, Contrastive Reinforcement Learning (CRL) provides a framework for policy updates using an approximation of the value function estimated via contrastive learning, achieving higher sample efficiency compared to conventional methods. However, since CRL treats the visited state as a pseudo-goal during learning, it can accurately estimate the value function only for limited goals. To address this issue, we propose a novel data augmentation approach for CRL called ViSA (Visited-State Augmentation). ViSA consists of two components: 1) generating augmented state samples, with the aim of augmenting hard-to-visit state samples during on-policy exploration, and 2) learning consistent embedding space, which uses an augmented state as auxiliary information to regularize the embedding space by reformulating the objective function of the embedding space based on mutual information. We evaluate ViSA in simulation and real-world robotic tasks and show improved goal-space generalization, which permits accurate value estimation for hard-to-visit goals. Further details can be found on the project page: https://issa-n.github.io/projectPage_ViSA/