Data to 7 Oct 2026

Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions

arXiv 1912.02875 ↗First posted 5 Dec 2019cs.AI cs.LGcited by 124

Not on the map. Why?

Cited by 43 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high rewards or other goals, through input commands such as: get lots of reward within at most so much time! A separate paper [63] on first experiments with UDRL shows that even a pilot version of UDRL can outperform traditional baseline algorithms on certain challenging RL problems. We also also conceptually simplify an approach [60] for teaching a robot to imitate humans. First videotape humans imitating the robot's current behaviors, then let the robot learn through SL to map the videos (as input commands) to these behaviors, then let it generalize and imitate videos of humans executing previously unknown behavior. This Imitate-Imitator concept may actually explain why biological evolution has resulted in parents who imitate the babbling of their babies.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 26; 2023: 38; 2024: 22; 2025: 22; 2026: 9.

124 papers.

Sort:

By area:

2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
2021GFlowNet Foundationscited by 286
2025π*0.6: a VLA That Learns From Experiencecited by 280
2022The Alignment Problem from a Deep Learning Perspectivecited by 246
2023Pretraining Language Models with Human Preferencescited by 245
2022Multi-Game Decision Transformerscited by 212
2022Online Decision Transformercited by 194
2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145

38 more

2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
2023Synthetic Experience Replaycited by 100
2023Guided Flows for Generative Modeling and Decision Makingcited by 99
2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
2022Efficient Planning in a Compact Latent Action Spacecited by 57
2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
2022Generative Pretraining for Black-Box Optimizationcited by 36
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
2023Learning to Modulate pre-trained Models in RLcited by 28
2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
2025Latent Diffusion Planning for Imitation Learningcited by 26
2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
2022Pareto Conditioned Networkscited by 22
2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
2023Passive learning of active causal strategies in agents and language modelscited by 21
2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20
2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
2025Fast Adaptation with Behavioral Foundation Modelscited by 20
2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
2025Curating Demonstrations using Online Experiencecited by 16
2023Learning Video-Conditioned Policies for Unseen Manipulation Taskscited by 15

50 most cited of 124 All 124 (JSON)