mlsanityData to 7 Oct 2026

Full-Gradient Successor Feature Representations

arXiv 2604.00686 ↗First posted 1 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Successor Features (SF) combined with Generalized Policy Improvement (GPI) provide a robust framework for transfer learning in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. However, standard SF learning methods typically rely on semi-gradient Temporal Difference (TD) updates. When combined with non-linear function approximation, semi-gradient methods lack robust convergence guarantees and can lead to instability, particularly in the multi-task setting where accurate feature estimation is critical for effective GPI. Inspired by Full Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), an algorithm that optimizes the successor features by minimizing the full Mean Squared Bellman Error. Unlike standard approaches, our method computes gradients with respect to parameters in both the online and target networks. We provide a theoretical proof of almost-sure convergence for FG-SFRQL and demonstrate empirically that minimizing the full residual leads to superior sample efficiency and transfer performance compared to semi-gradient baselines in both discrete and continuous domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

16 of 29 references.

References · 16

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  3. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  4. 2021APS: Active Pretraining with Successor Featurescited by 109
  5. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  6. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  7. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  8. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  9. 2021Full Gradient DQN Reinforcement Learning: A Provably Convergent Schemecited by 8
  10. 2019Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learningcited by 6
  11. 2021Successor Feature Representationscited by 6
  12. 2024SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learningcited by 5
4 more
  1. 2016Stochastic Recursive Inclusions with Non-Additive Iterate-Dependent Markov Noisecited by 3
  2. 2025SR-Reward: Taking The Path More Traveledcited by 3
  3. 2022Safety-Constrained Policy Transfer with Successor Featurescited by 2
  4. 2024Decoupling Exploration and Exploitation for Unsupervised Pre-training with Successor Featurescited by 1

Built on by

Not cited yet.