# Learning from Mixed-Quality Deployment Experience for Robot Manipulation

arXiv: https://arxiv.org/abs/2609.29000 · HTML: https://mlsanity.com/paper/2609.29000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 28 of 28)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies](https://mlsanity.com/paper/2304.10573/) (2023)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Interactive Post-Training for Vision-Language-Action Models](https://mlsanity.com/paper/2505.17016/) (2025)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance](https://mlsanity.com/paper/2410.13816/) (2024)
- [Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2207.10050/) (2022)
- [RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning](https://mlsanity.com/paper/2510.14830/) (2025)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction](https://mlsanity.com/paper/2509.07953/) (2025)
- [So You Think You Can Scale Up Autonomous Robot Data Collection?](https://mlsanity.com/paper/2411.01813/) (2024)
- [Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning](https://mlsanity.com/paper/2310.14196/) (2023)
- [Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies](https://mlsanity.com/paper/2605.00416/) (2026)
- [Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulation](https://mlsanity.com/paper/2401.08957/) (2024)
- [Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Reward](https://mlsanity.com/paper/2508.11143/) (2025)
- [How to Leverage Diverse Demonstrations in Offline Imitation Learning](https://mlsanity.com/paper/2405.17476/) (2024)
- [Improving Behavioural Cloning with Positive Unlabeled Learning](https://mlsanity.com/paper/2301.11734/) (2023)
- [Towards Effective Utilization of Mixed-Quality Demonstrations in Robotic Manipulation via Segment-Level Selection and Optimization](https://mlsanity.com/paper/2409.19917/) (2024)
- [Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning](https://mlsanity.com/paper/2510.19495/) (2025)
- [WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning](https://mlsanity.com/paper/2607.29613/) (2026)

## Abstract (arXiv)

Robot policies deployed in real environments naturally accumulate mixed-quality experience, including successful executions, partial progress, and failures. Although these rollouts provide valuable information for further learning, directly incorporating them into imitation learning may reinforce undesirable behaviors, while offline reinforcement learning often suffers from unreliable value estimation under sparse rewards and limited data coverage. We consider a practical post-deployment setting where learning relies only on naturally accumulated autonomous rollouts, without additional human corrections or exploratory interaction. To effectively exploit such experience, we propose Predictive Action Chunk Learning (PACL). PACL first learns a predictive chunk-level critic that evaluates temporally extended action sequences and augments temporal difference learning with future latent prediction, providing richer supervision for long-horizon value estimation. The learned critic then converts chunk-level Q-values into discrete quality conditions, which guide a diffusion actor to learn jointly from these mixed-quality experiences without treating all behaviors as equivalent supervision. At inference, the actor generates multiple action chunks and the critic selects the highest valued candidate. Experiments across simulated and real-world robot manipulation tasks show that PACL consistently improves the pretrained policy and outperforms strong imitation learning and offline reinforcement learning baselines.