mlsanityData to 7 Oct 2026

Optimizing Agent Behavior over Long Time Scales by Transporting Value

arXiv 1810.06721 ↗First posted 15 Oct 2018cs.AI cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

Humans spend a remarkable fraction of waking life engaged in acts of "mental time travel". We dwell on our actions in the past and experience satisfaction or regret. More than merely autobiographical storytelling, we use these event recollections to change how we will act in similar scenarios in the future. This process endows us with a computationally important ability to link actions and consequences across long spans of time, which figures prominently in addressing the problem of long-term temporal credit assignment; in artificial intelligence (AI) this is the question of how to evaluate the utility of the actions within a long-duration behavioral sequence leading to success or failure in a task. Existing approaches to shorter-term credit assignment in AI cannot solve tasks with long delays between actions and consequences. Here, we introduce a new paradigm for reinforcement learning where agents use recall of specific memories to credit actions from the past, allowing them to solve problems that are intractable for existing algorithms. This paradigm broadens the scope of problems that can be investigated in AI and offers a mechanistic account of behaviors that may inspire computational models in neuroscience, psychology, and behavioral economics.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 10; 2023: 11; 2024: 11; 2025: 7; 2026: 5.

57 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  3. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  4. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  5. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  6. 2024Mastering Memory Tasks with World Modelscited by 38
  7. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  8. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  9. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  10. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  11. 2024Equivariant Reinforcement Learning under Partial Observabilitycited by 17
  12. 2021Learning Reward Machines: A Study in Partially Observable Reinforcement Learningcited by 16
38 more
  1. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  2. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  3. 2022Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomescited by 11
  4. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  5. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  6. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  7. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  8. 2024Rethinking Transformers in Solving POMDPscited by 9
  9. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learningcited by 8
  10. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  11. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  12. 2024Token-level Proximal Policy Optimization for Query Generationcited by 7
  13. 2021Preferential Temporal Difference Learningcited by 6
  14. 2021Towards Practical Credit Assignment for Deep Reinforcement Learningcited by 5
  15. 2021Episodic Policy Gradient Trainingcited by 5
  16. 2024Spatially-Aware Transformer for Embodied Agentscited by 5
  17. 2024VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Makingcited by 5
  18. 2022Selective Credit Assignmentcited by 4
  19. 2022Contrastive Retrospection: honing in on critical steps for rapid learning and generalization in RLcited by 4
  20. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
  21. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  22. 2022Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Gamescited by 3
  23. 2024Action abstractions for amortized samplingcited by 3
  24. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  25. 2021Adaptive Pairwise Weights for Temporal Credit Assignmentcited by 2
  26. 2023Grid Cell-Inspired Fragmentation and Recall for Efficient Map Buildingcited by 2
  27. 2023Neuro-Inspired Fragmentation and Recall to Overcome Catastrophic Forgetting in Curiositycited by 2
  28. 2023Episodic Return Decomposition by Difference of Implicitly Assigned Sub-Trajectory Rewardcited by 2
  29. 2021Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific researchcited by 1
  30. 2022Bayesian sense of time in biological and artificial brainscited by 1
  31. 2022Modeling Human Behavior Part I -- Learning and Belief Approachescited by 1
  32. 2022Off-Beat Multi-Agent Reinforcement Learningcited by 1
  33. 2023PCGPT: Procedural Content Generation via Transformerscited by 1
  34. 2025The challenge of hidden gifts in multi-agent reinforcement learningcited by 1
  35. 2025Agentic Episodic Controlcited by 1
  36. 2026Reinforcing the World's Edge: A Continual Learning Problem in the Multi-Agent-World Boundarycited by 1
  37. 2021Delayed Rewards Calibration via Reward Empirical Sufficiencycited by 0
  38. 2022Multi-Agent Reinforcement Learning for Long-Term Network Resource Allocation through Auction: a V2X Applicationcited by 0

50 most cited of 57 All 57 (JSON)