mlsanityData to 14 Oct 2026

Deep Intrinsic Surprise-Regularized Control (DISRC): A Biologically Inspired Mechanism for Efficient Deep Q-Learning in Sparse Environments

arXiv 2601.17598 ↗First posted 24 Jan 2026cs.NE cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning (DRL) has driven major advances in autonomous control. Still, standard Deep Q-Network (DQN) agents tend to rely on fixed learning rates and uniform update scaling, even as updates are modulated by temporal-difference (TD) error. This rigidity destabilizes convergence, especially in sparse-reward settings where feedback is infrequent. We introduce Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired augmentation to DQN that dynamically scales Q-updates based on latent-space surprise. DISRC encodes states via a LayerNorm-based encoder and computes a deviation-based surprise score relative to a moving latent setpoint. Each update is then scaled in proportion to both TD error and surprise intensity, promoting plasticity during early exploration and stability as familiarity increases. We evaluate DISRC on two sparse-reward MiniGrid environments, which included MiniGrid-DoorKey-8x8 and MiniGrid-LavaCrossingS9N1, under identical settings as a vanilla DQN baseline. In DoorKey, DISRC reached the first successful episode (reward > 0.8) 33% faster than the vanilla DQN baseline (79 vs. 118 episodes), with lower reward standard deviation (0.25 vs. 0.34) and higher reward area under the curve (AUC: 596.42 vs. 534.90). These metrics reflect faster, more consistent learning - critical for sparse, delayed reward settings. In LavaCrossing, DISRC achieved a higher final reward (0.95 vs. 0.93) and the highest AUC of all agents (957.04), though it converged more gradually. These preliminary results establish DISRC as a novel mechanism for regulating learning intensity in off-policy agents, improving both efficiency and stability in sparse-reward domains. By treating surprise as an intrinsic learning signal, DISRC enables agents to modulate updates based on expectation violations, enhancing decision quality when conventional value-based methods fall short.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 6 of its 12 references that are in the map. First posted on arXiv 24 Jan 2026.

Builds on

6 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Episodic Reinforcement Learning with Expanded State-reward Space19 Jan

Other references · 5

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2025A Survey on Explainable Deep Reinforcement Learningcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
  2. 2025β-DQN: Improving Deep Q-Learning By Evolving the Behavior
  3. 2026Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning
  4. 2026Deep Q-Learning on H"older Spaces
  5. 2024Self-evolving Autoencoder Embedded Q-Network
  6. 2026GIFT: Global stabilisation via Intrinsic Fine Tuning
  7. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
  8. 2025Universal Approximation Theorem of Deep Q-Networks