mlsanityData to 14 Oct 2026

Rank-Then-Act: Reward-Free Control from Frame-Order Progress

arXiv 2607.01897 ↗First posted 2 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains a Vision-Language Model (VLM) offline as a progress-based ordinal scorer, using a Group Relative Policy Optimization (GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function for reinforcement learning: at each interaction window, we compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms prior video-based reward learning methods and rank-based baselines, while demonstrating strong cross-task reuse of a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 29 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

26 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  8. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  10. 2018Generative Adversarial Imitation from Observationcited by 184
  11. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  12. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
13 more
  1. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  2. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  3. 2024Vision Language Models are In-Context Value Learnerscited by 90
  4. 2020Adaptive Reward-Free Explorationcited by 77
  5. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  6. 2021Learning by Aligning Videos in Timecited by 48
  7. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 43
  8. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 36
  9. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 32
  10. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  11. 2024VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulationcited by 12
  12. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  13. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Successcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
  2. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement
  3. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
  4. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learning
  5. 2026Disentangling Task-Relevant Latent Dynamics for Reward Modeling
  6. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
  7. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
  8. 2025Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning