# Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation

arXiv: https://arxiv.org/abs/2102.05714 · HTML: https://mlsanity.com/paper/2102.05714/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [Model-free Deep Reinforcement Learning for Urban Autonomous Driving](https://mlsanity.com/paper/1904.09503/) (2019)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation](https://mlsanity.com/paper/1806.07377/) (2018)
- [Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders](https://mlsanity.com/paper/1804.10469/) (2018)
- [Adapting Deep Visuomotor Representations with Weak Pairwise Constraints](https://mlsanity.com/paper/1511.07111/) (2015)

## Built on by (most cited first, 30 of 35)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation](https://mlsanity.com/paper/2411.09891/) (2024)
- [MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2312.15339/) (2023)
- [Beyond the Edge: An Advanced Exploration of Reinforcement Learning for Mobile Edge Computing, its Applications, and Future Research Trajectories](https://mlsanity.com/paper/2404.14238/) (2024)
- [Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics](https://mlsanity.com/paper/2505.13150/) (2025)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator](https://mlsanity.com/paper/2509.08221/) (2025)
- [No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL](https://mlsanity.com/paper/2205.08716/) (2022)
- [Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents](https://mlsanity.com/paper/2412.11484/) (2024)
- [Domain Generalization for Vision-based Driving Trajectory Generation](https://mlsanity.com/paper/2109.13858/) (2021)
- [Opportunities and Challenges from Using Animal Videos in Reinforcement Learning for Navigation](https://mlsanity.com/paper/2209.12347/) (2022)
- [Online Prototype Alignment for Few-shot Policy Transfer](https://mlsanity.com/paper/2306.07307/) (2023)
- [A Framework for Few-Shot Policy Transfer through Observation Mapping and Behavior Cloning](https://mlsanity.com/paper/2310.08836/) (2023)
- [Automaton Distillation: Neuro-Symbolic Transfer Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2310.19137/) (2023)
- [Offline Reinforcement Learning with Domain-Unlabeled Data](https://mlsanity.com/paper/2404.07465/) (2024)
- [Depth Transfer: Learning to See Like a Simulator for Real-World Drone Navigation](https://mlsanity.com/paper/2505.12428/) (2025)
- [P3O: Transferring Visual Representations for Reinforcement Learning via Prompting](https://mlsanity.com/paper/2303.12371/) (2023)
- [Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments](https://mlsanity.com/paper/2506.21129/) (2025)
- [Malleable Agents for Re-Configurable Robotic Manipulators](https://mlsanity.com/paper/2202.02395/) (2022)
- [Model Generation with Provable Coverability for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.00316/) (2022)
- [Unified State Representation Learning under Data Augmentation](https://mlsanity.com/paper/2209.05302/) (2022)
- [Zero-Shot Transfer in Imitation Learning](https://mlsanity.com/paper/2310.06710/) (2023)
- [Imitation learning for sim-to-real transfer of robotic cutting policies based on residual Gaussian process disturbance force model](https://mlsanity.com/paper/2311.04096/) (2023)
- [Automatic re-calibration of quantum devices by reinforcement learning](https://mlsanity.com/paper/2404.10726/) (2024)
- [Prompt-based Visual Alignment for Zero-shot Policy Transfer](https://mlsanity.com/paper/2406.03250/) (2024)
- [Planning Under Observation Mismatch for Traffic Signal Control via Adaptive Modular World Models](https://mlsanity.com/paper/2501.02548/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Reinforcement Learning Driven Generalizable Feature Representation for Cross-User Activity Recognition](https://mlsanity.com/paper/2509.01031/) (2025)
- [DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation](https://mlsanity.com/paper/2509.04970/) (2025)
- [Vehicle Dynamics Embedded World Models for Autonomous Driving](https://mlsanity.com/paper/2512.02417/) (2025)

## Abstract (arXiv)

Despite the recent success of deep reinforcement learning (RL), domain adaptation remains an open problem. Although the generalization ability of RL agents is critical for the real-world applicability of Deep RL, zero-shot policy transfer is still a challenging problem since even minor visual changes could make the trained agent completely fail in the new task. To address this issue, we propose a two-stage RL agent that first learns a latent unified state representation (LUSR) which is consistent across multiple domains in the first stage, and then do RL training in one source domain based on LUSR in the second stage. The cross-domain consistency of LUSR allows the policy acquired from the source domain to generalize to other target domains without extra training. We first demonstrate our approach in variants of CarRacing games with customized manipulations, and then verify it in CARLA, an autonomous driving simulator with more complex and realistic visual observations. Our results show that this approach can achieve state-of-the-art domain adaptation performance in related RL tasks and outperforms prior approaches based on latent-representation based RL and image-to-image translation.