mlsanityData to 7 Oct 2026

Beyond Surprise: Improving Exploration Through Surprise Novelty

arXiv 2308.04836 ↗First posted 9 Aug 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations. The reward is the novelty of the surprise rather than the surprise norm. We estimate the surprise novelty as retrieval errors of a memory network wherein the memory stores and reconstructs surprises. Our surprise memory (SM) augments the capability of surprise-based intrinsic motivators, maintaining the agent's interest in exciting exploration while reducing unwanted attraction to unpredictable or noisy observations. Our experiments demonstrate that the SM combined with various surprise predictors exhibits efficient exploring behaviors and significantly boosts the final performance in sparse reward environments, including Noisy-TV, navigation and challenging Atari games.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

19 of 34 references.

References · 19

  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2014Neural Turing Machinescited by 630
  5. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  7. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  8. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  9. 2019Self-Supervised Exploration via Disagreementcited by 263
  10. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  11. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  12. 2018Episodic Curiosity through Reachabilitycited by 109
7 more
  1. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  2. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  3. 2020Latent World Models For Intrinsically Motivated Explorationcited by 15
  4. 2019Learning to Remember More with Less Memorizationcited by 14
  5. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  6. 2022Learning to Constrain Policy Optimization with Virtual Trust Regioncited by 4
  7. 2022Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learningcited by 8
  2. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  3. 2025Action-Dependent Optimality-Preserving Reward Shapingcited by 2
  4. 2025Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviorscited by 0
  5. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learningcited by 0