mlsanityData to 14 Oct 2026

Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning

arXiv 2608.17373 ↗First posted 18 Aug 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 47 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

30 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method30 Jan
  2. 2024Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks31 Jul

Other references · 28

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2015Prioritized Experience Replaycited by 1,099
  6. 2018DeepMind Control Suitecited by 956
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  8. 2017Deep Reinforcement Learning that Matterscited by 754
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  11. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  12. 2020Revisiting Fundamentals of Experience Replaycited by 175
16 more
  1. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  2. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  3. 2019A survey on intrinsic motivation in reinforcement learningcited by 81
  4. 2019Experience Replay Optimizationcited by 48
  5. 2019Curiosity-Driven Experience Prioritization via Density Estimationcited by 27
  6. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  7. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  8. 2022Actor Prioritized Experience Replaycited by 20
  9. 2020Understanding and Mitigating the Limitations of Prioritized Experience Replaycited by 17
  10. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  11. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  12. 2025Uncertainty Prioritized Experience Replaycited by 6
  13. 2023Beyond Surprise: Improving Exploration Through Surprise Noveltycited by 5
  14. 2019Visual novelty, curiosity, and intrinsic reward in machine learning and the braincited by 4
  15. 2020ACDER: Augmented Curiosity-Driven Experience Replaycited by 4
  16. 2022CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating the Interplay of Prioritized Replay and Generalization
  2. 2025Reliability-Adjusted Prioritized Experience Replay
  3. 2024Prioritized Generative Replay
  4. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  5. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learning
  6. 2023Curious Replay for Model-based Adaptation
  7. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms
  8. 2025Sample Efficient Experience Replay in Non-stationary Environments