mlsanityData to 14 Oct 2026

SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewards

arXiv 2609.35268 ↗First posted 28 Sep 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) with sparse rewards is challenging because delayed outcomes provide little guidance about which intermediate computations caused success or failure. We argue that reliable credit assignment requires policy dynamics that preserve and expose credit-relevant information over time, a role we formalize as Temporal Credit Carriers (TCCs) and that spiking neural networks (SNNs) naturally fulfill through graded membrane traces and event-driven spikes. Based on this hypothesis, we propose SpikeCredit, an SNN-based framework for RL with sparse rewards that first performs task-adaptive TCC selection and then closes the loop between a fast TCC-reading pathway, where self-motion feedback constraint uses local behavior-grounded cues to constrain transition-level credit recovery, and a slow TCC-writing pathway, where credit-targeted trace alignment feeds recovered credit back into the actor to make future TCC dynamics more credit-readable. Across sparse-reward MuJoCo tasks, SpikeCredit improves Last10 return over sparse SNN baselines by +1169% on Ant, +953% on Hopper, +723% on Swimmer, and +1781% on Walker2d, and exceeds the dense-reward baseline on Swimmer by +113%. Mechanistic analyses further show substantially stronger alignment with dense rewards than the sparse SNN baseline. These results position spiking dynamics as credit-preserving substrates for sparse-reward RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 43 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

20 of 43 references.

References · 20

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2016OpenAI Gymcited by 1,736
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  7. 2019Self-Supervised Exploration via Disagreementcited by 265
  8. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  9. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  10. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  11. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  12. 2019Hindsight Credit Assignmentcited by 79
8 more
  1. 2021Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problemscited by 26
  2. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  3. 2020Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networkscited by 23
  4. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  5. 2025Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Controlcited by 6
  6. 2024GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARLcited by 3
  7. 2025CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learningcited by 3
  8. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning
  2. 2021InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem
  3. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RL
  4. 2024AdaCred: Adaptive Causal Decision Transformers with Feature Crediting
  5. 2024Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning
  6. 2025Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
  7. 2026Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents
  8. 2026Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning