mlsanityData to 7 Oct 2026

Episodic Novelty Through Temporal Distance

arXiv 2501.15418 ↗First posted 26 Jan 2025cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Exploration in sparse reward environments remains a significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrinsic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance as a robust metric for state similarity and intrinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-art methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 26 Jan 2025.

Builds on

52 of 63 references.

References · 52

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018DeepMind Control Suitecited by 935
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  8. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  12. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
38 more
  1. 2017Count-Based Exploration with Neural Density Modelscited by 255
  2. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  4. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  5. 2020The NetHack Learning Environmentcited by 196
  6. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  7. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  8. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  9. 2017Neural Episodic Controlcited by 148
  10. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  11. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  12. 2018Count-Based Exploration with the Successor Representationcited by 112
  13. 2018Episodic Curiosity through Reachabilitycited by 109
  14. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  15. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  16. 2016Model-Free Episodic Controlcited by 100
  17. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  18. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  19. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  20. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  21. 2022Exploration via Elliptical Episodic Bonusescited by 51
  22. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  23. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
  24. 2021Adversarially Guided Actor-Criticcited by 47
  25. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  26. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  27. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  28. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  29. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  30. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  31. 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 22
  32. 2023Explore to Generalize in Zero-Shot RLcited by 20
  33. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  34. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  35. 2022Reachability-Aware Laplacian Representation in Reinforcement Learningcited by 10
  36. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  37. 2022An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environmentscited by 8
  38. 2023DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewardscited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
  2. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 5
  3. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  4. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  5. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  6. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVscited by 1
  7. 2025Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoringcited by 1
  8. 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewardscited by 1
  9. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  10. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  11. 2025Inter-Agent Relative Representations for Multi-Agent Option Discoverycited by 0
  12. 2026SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learningcited by 0
3 more
  1. 2026Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Controlcited by 0
  2. 2026Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learningcited by 0
  3. 2026CIG: Exploration via Conditional Information Gaincited by 0