# Learning from Visual Observation via Offline Pretrained State-to-Go Transformer

arXiv: https://arxiv.org/abs/2306.12860 · HTML: https://mlsanity.com/paper/2306.12860/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Reinforcement and Imitation Learning for Diverse Visuomotor Skills](https://mlsanity.com/paper/1802.09564/) (2018)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Imitating Latent Policies from Observation](https://mlsanity.com/paper/1805.07914/) (2018)
- [Off-Policy Imitation Learning from Observations](https://mlsanity.com/paper/2102.13185/) (2021)
- [MobILE: Model-Based Imitation Learning From Observation Alone](https://mlsanity.com/paper/2102.10769/) (2021)
- [An Optimistic Perspective on Offline Reinforcement Learning](https://mlsanity.com/paper/1907.04543/) (2019)
- [Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks](https://mlsanity.com/paper/2303.16563/) (2023)
- [Visual Imitation Learning with Patch Rewards](https://mlsanity.com/paper/2302.00965/) (2023)
- [Reinforcement Learning Friendly Vision-Language Model for Minecraft](https://mlsanity.com/paper/2303.10571/) (2023)
- [Imitation Learning from Observations under Transition Model Disparity](https://mlsanity.com/paper/2204.11446/) (2022)
- [Guiding Online Reinforcement Learning with Action-Free Offline Pretraining](https://mlsanity.com/paper/2301.12876/) (2023)
- [Imitating Unknown Policies via Exploration](https://mlsanity.com/paper/2008.05660/) (2020)

## Built on by (most cited first, 14 of 14)

- [Vision-Language Models Provide Promptable Representations for Reinforcement Learning](https://mlsanity.com/paper/2402.02651/) (2024)
- [HarmonyDream: Task Harmonization Inside World Models](https://mlsanity.com/paper/2310.00344/) (2023)
- [MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory](https://mlsanity.com/paper/2411.06736/) (2024)
- [MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation](https://mlsanity.com/paper/2505.16602/) (2025)
- [Robot Policy Learning with Temporal Optimal Transport Reward](https://mlsanity.com/paper/2410.21795/) (2024)
- [Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers](https://mlsanity.com/paper/2410.24108/) (2024)
- [Pre-trained Visual Dynamics Representations for Efficient Policy Learning](https://mlsanity.com/paper/2411.03169/) (2024)
- [FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making](https://mlsanity.com/paper/2507.12496/) (2025)
- [CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Tasks](https://mlsanity.com/paper/2508.18797/) (2025)
- [Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos](https://mlsanity.com/paper/2412.10778/) (2024)
- [NOLO: Navigate Only Look Once](https://mlsanity.com/paper/2408.01384/) (2024)
- [Action-Free Offline-to-Online RL via Discretised State Policies](https://mlsanity.com/paper/2602.00629/) (2026)
- [Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization](https://mlsanity.com/paper/2607.00796/) (2026)
- [From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training](https://mlsanity.com/paper/2607.00811/) (2026)

## Abstract (arXiv)

Learning from visual observation (LfVO), aiming at recovering policies from only visual observation data, is promising yet a challenging problem. Existing LfVO approaches either only adopt inefficient online learning schemes or require additional task-specific information like goal states, making them not suited for open-ended tasks. To address these issues, we propose a two-stage framework for learning from visual observation. In the first stage, we introduce and pretrain State-to-Go (STG) Transformer offline to predict and differentiate latent transitions of demonstrations. Subsequently, in the second stage, the STG Transformer provides intrinsic rewards for downstream reinforcement learning tasks where an agent learns merely from intrinsic rewards. Empirical results on Atari and Minecraft show that our proposed method outperforms baselines and in some tasks even achieves performance comparable to the policy learned from environmental rewards. These results shed light on the potential of utilizing video-only data to solve difficult visual reinforcement learning tasks rather than relying on complete offline datasets containing states, actions, and rewards. The project's website and code can be found at https://sites.google.com/view/stgtransformer.