# Reinforcement Learning with Latent Flow

arXiv: https://arxiv.org/abs/2101.01857 · HTML: https://mlsanity.com/paper/2101.01857/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 58 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)

## Built on by (most cited first, 19 of 19)

- [The Unsurprising Effectiveness of Pre-Trained Vision Models for Control](https://mlsanity.com/paper/2203.03580/) (2022)
- [Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning](https://mlsanity.com/paper/2212.08860/) (2022)
- [On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline](https://mlsanity.com/paper/2212.05749/) (2022)
- [VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2202.10324/) (2022)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents](https://mlsanity.com/paper/2309.17207/) (2023)
- [State-Wise Safe Reinforcement Learning With Pixel Observations](https://mlsanity.com/paper/2311.02227/) (2023)
- [When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?](https://mlsanity.com/paper/2412.13662/) (2024)
- [RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learning](https://mlsanity.com/paper/2401.14858/) (2024)
- [The Temporal Trap: Entanglement in Pre-Trained Visual Representations for Visuomotor Policy Learning](https://mlsanity.com/paper/2502.03270/) (2025)
- [DyMoDreamer: World Modeling with Dynamic Modulation](https://mlsanity.com/paper/2509.24804/) (2025)
- [ResWM: Residual-Action World Model for Visual RL](https://mlsanity.com/paper/2603.11110/) (2026)
- [COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control](https://mlsanity.com/paper/2601.06122/) (2026)
- [Temporal Shift Reinforcement Learning](https://mlsanity.com/paper/2109.02145/) (2021)
- [Weakly Supervised Scene Text Detection using Deep Reinforcement Learning](https://mlsanity.com/paper/2201.04866/) (2022)
- [CRC-RL: A Novel Visual Feature Representation Architecture for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2301.13473/) (2023)
- [Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Control](https://mlsanity.com/paper/2406.06072/) (2024)
- [Learn 2 Rage: Experiencing The Emotional Roller Coaster That Is Reinforcement Learning](https://mlsanity.com/paper/2410.18462/) (2024)
- [Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning](https://mlsanity.com/paper/2512.05172/) (2025)

## Abstract (arXiv)

Temporal information is essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information is given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning (Flare), a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare (i) recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information, (ii) achieves state-of-the-art performance on pixel-based challenging continuous control tasks within the DeepMind control benchmark suite, namely quadruped walk, hopper hop, finger turn hard, pendulum swing, and walker run, and is the most sample efficient model-free pixel-based RL algorithm, outperforming the prior model-free state-of-the-art by 1.9X and 1.5X on the 500k and 1M step benchmarks, respectively, and (iv), when augmented over rainbow DQN, outperforms this state-of-the-art level baseline on 5 of 8 challenging Atari games at 100M time step benchmark.