# ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies

arXiv: https://arxiv.org/abs/2608.02958 · HTML: https://mlsanity.com/paper/2608.02958/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models.
- Builds on 42 of its 48 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Placed in

- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)

## Builds on (most cited first, 30 of 42)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [Interactive Language: Talking to Robots in Real Time](https://mlsanity.com/paper/2210.06407/) (2022)

## Built on by (most cited first, 1 of 1)

- [Skill-Space Shooting for Autonomous Robot Policy Improvement](https://mlsanity.com/paper/2609.38178/) (2026)

## Abstract (arXiv)

Generalist vision-language-action (VLA) policies now transfer to new scenes with little task-specific training, but they still fail silently: from the action stream alone, a collapsing rollout looks much like one making clean progress, because imitation supplies no notion of progress. The cheap alternative, a terminal success / failure bit, is learnable in principle yet too sparse to say when a rollout went wrong. We argue that the per-frame label is the hard part, and that it works better when continuous and non-monotonic: a score rather than a single success or failure, and one free to fall and then rise again as a rollout goes wrong and recovers. Such a label is cheap to obtain: a person marks only the stage at which an episode went wrong, and the per-frame values are computed from that. We present ValueFormer, a policy-agnostic causal transformer (3.5M parameters) over a frozen DINOv3 backbone, emitting two per-frame signals in one forward pass: a smooth Monte Carlo value for advantage estimation and a sharp binary value for online mistake detection. Failed episodes carry a stage-aware, success-then-decay return. A teleoperation station already records an intervention flag; reusing it lifts held-out detection from 0.38 to 0.82 average precision and flags 95% of mistakes before the operator reacts. On a real bimanual sandwich station, feeding the critic back as a per-frame post-training weight raises task completion from 70% to 85% and removes a repeat-pick failure mode in our trials, although at n=20 that gain sits within noise.