mlsanityData to 14 Oct 2026

ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulation

arXiv 2509.22402 ↗First posted 26 Sep 2025cs.LG cs.ROcited by 1

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Reward design remains a critical bottleneck in visual reinforcement learning (RL) for robotic manipulation. In simulated environments, rewards are conventionally designed based on the distance to a target position. However, such precise positional information is often unavailable in real-world visual settings due to sensory and perceptual limitations. In this study, we propose a method that implicitly infers spatial distances through keypoints extracted from images. Building on this, we introduce Reward Learning with Anticipation Model (ReLAM), a novel framework that automatically generates dense, structured rewards from action-free video demonstrations. ReLAM first learns an anticipation model that serves as a planner and proposes intermediate keypoint-based subgoals on the optimal path to the final goal, creating a structured learning curriculum directly aligned with the task's geometric objectives. Based on the anticipated subgoals, a continuous reward signal is provided to train a low-level, goal-conditioned policy under the hierarchical reinforcement learning (HRL) framework with provable sub-optimality bound. Extensive experiments on complex, long-horizon manipulation tasks show that ReLAM significantly accelerates learning and achieves superior performance compared to state-of-the-art methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 40 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

30 of 40 references.

References · 30

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  11. 2023CoTracker: It is Better to Track Togethercited by 522
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
18 more
  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  2. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  3. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  4. 2024Diffusion Policy Policy Optimizationcited by 280
  5. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  6. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  7. 2018Generative Adversarial Imitation from Observationcited by 184
  8. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  10. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  11. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 128
  12. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  13. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  14. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 57
  15. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  16. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  17. 2022Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemblecited by 5
  18. 2025Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Taskscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
  2. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
  3. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning
  4. 2024Reward Machine Inference for Robotic Manipulation
  5. 2026Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
  6. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
  7. 2026RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards ↗
  8. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation