mlsanityData to 7 Oct 2026

RUDDER: Return Decomposition for Delayed Rewards

arXiv 1806.07857 ↗First posted 20 Jun 2018cs.LG cs.AI math.OCcited by 162

Not on the map. Why?

Cited by 31 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance problems in Monte Carlo (MC) learning. Both problems are even more severe when rewards are delayed. RUDDER aims at making the expected future rewards zero, which simplifies Q-value estimation to computing the mean of the immediate reward. We propose the following two new concepts to push the expected future rewards toward zero. (i) Reward redistribution that leads to return-equivalent decision processes with the same optimal policies and, when optimal, zero expected future rewards. (ii) Return decomposition via contribution analysis which transforms the reinforcement learning task into a regression task at which deep learning excels. On artificial tasks with delayed rewards, RUDDER is significantly faster than MC and exponentially faster than Monte Carlo Tree Search (MCTS), TD(λ), and reward shaping approaches. At Atari games, RUDDER on top of a Proximal Policy Optimization (PPO) baseline improves the scores, which is most prominent at games with delayed rewards. Source code is available at urlhttps://github.com/ml-jku/rudder and demonstration videos at urlhttps://goo.gl/EQerZV.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 18; 2022: 18; 2023: 26; 2024: 21; 2025: 25; 2026: 53.

162 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  3. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  4. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  5. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  6. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  7. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  8. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  9. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  10. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  11. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
  12. 2024Mastering Memory Tasks with World Modelscited by 38
38 more
  1. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  2. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  3. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  4. 2023Learning to Modulate pre-trained Models in RLcited by 28
  5. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  6. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  7. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  8. 2023Learning Energy Decompositions for Partial Inference of GFlowNetscited by 24
  9. 2026TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMscited by 24
  10. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 23
  11. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  12. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  13. 2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
  14. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  15. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  16. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  17. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  18. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  19. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  20. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  21. 2023A Survey of Demonstration Learningcited by 12
  22. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  23. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  24. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 11
  25. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  26. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  27. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistributioncited by 10
  28. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  29. 2022Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learningcited by 9
  30. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9
  31. 2023Learning to Identify Critical States for Reinforcement Learning from Videoscited by 9
  32. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  33. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  34. 2021Search in Imperfect Information Gamescited by 8
  35. 2022Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 8
  36. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8
  37. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 8
  38. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7

50 most cited of 162 All 162 (JSON)