mlsanityData to 14 Oct 2026

ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

arXiv 2610.18596 ↗First posted 13 Oct 2026cs.RO cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation also fits: Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0% to 68.5%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 43 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

39 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning22 Aug
  2. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models25 May
  3. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Models24 Apr
  4. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutes1 Dec
  5. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  6. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control28 May
  7. 2025Improving Vision-Language-Action Model with Online Reinforcement Learning28 Jan

Other references · 32

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  8. 2025DINOv3cited by 1,414
  9. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 765
20 more
  1. 2026World Action Models are Zero-shot Policiescited by 465
  2. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  3. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  4. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  5. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 142
  6. 2025Reinforcement Learning with Action Chunkingcited by 107
  7. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  8. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  9. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 67
  10. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 53
  11. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 45
  12. 2024AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometriescited by 26
  13. 2024FORGE: Force-Guided Exploration for Robust Contact-Rich Manipulation under Uncertaintycited by 23
  14. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 22
  15. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learningcited by 17
  16. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  17. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  18. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 10
  19. 2026CRL-VLA: Continual Vision-Language-Action Learningcited by 7
  20. 2026Memory Anchors for Continual Robot Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Continual Policy Consolidation for Lifelong Robot Learning
  2. 2024Continual Task Learning through Adaptive Policy Self-Composition
  3. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
  4. 2026Can Vision-Language-Action Models Learn from Real-World Data Continually without Forgetting?
  5. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
  7. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  8. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning