# Inference-Time Policy Steering through Human Interactions

arXiv: https://arxiv.org/abs/2411.16627 · HTML: https://mlsanity.com/paper/2411.16627/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Goal-Conditioned Imitation Learning using Score-based Diffusion Policies](https://mlsanity.com/paper/2304.02532/) (2023)
- [Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC](https://mlsanity.com/paper/2302.11552/) (2023)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches](https://mlsanity.com/paper/2311.01977/) (2023)
- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance](https://mlsanity.com/paper/2410.13816/) (2024)
- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [PoCo: Policy Composition from and for Heterogeneous Robot Learning](https://mlsanity.com/paper/2402.02511/) (2024)
- [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024)
- [Compositional Diffusion-Based Continuous Constraint Solvers](https://mlsanity.com/paper/2309.00966/) (2023)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)

## Built on by (most cited first, 30 of 38)

- [DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control](https://mlsanity.com/paper/2502.05855/) (2025)
- [Reasoning with Sampling: Your Base Model is Smarter Than You Think](https://mlsanity.com/paper/2510.14901/) (2025)
- [RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models](https://mlsanity.com/paper/2506.17811/) (2025)
- [From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment](https://mlsanity.com/paper/2502.01828/) (2025)
- [DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance](https://mlsanity.com/paper/2506.13922/) (2025)
- [Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification](https://mlsanity.com/paper/2510.16281/) (2025)
- [VLS: Steering Pretrained Robot Policies via Vision-Language Models](https://mlsanity.com/paper/2602.03973/) (2026)
- [MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recovery](https://mlsanity.com/paper/2511.05007/) (2025)
- [Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning](https://mlsanity.com/paper/2506.00867/) (2025)
- [Latent Policy Steering with Embodiment-Agnostic Pretrained World Models](https://mlsanity.com/paper/2507.13340/) (2025)
- [Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusion](https://mlsanity.com/paper/2511.14178/) (2025)
- [Survey of Vision-Language-Action Models for Embodied Manipulation](https://mlsanity.com/paper/2508.15201/) (2025)
- [OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policies](https://mlsanity.com/paper/2603.10052/) (2026)
- [You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector](https://mlsanity.com/paper/2603.15757/) (2026)
- [Tree-Guided Diffusion Planner](https://mlsanity.com/paper/2508.21800/) (2025)
- [The Unreasonable Effectiveness of Discrete-Time Gaussian Process Mixtures for Robot Policy Learning](https://mlsanity.com/paper/2505.03296/) (2025)
- [Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondence](https://mlsanity.com/paper/2506.12678/) (2025)
- [Self-Guided Action Diffusion](https://mlsanity.com/paper/2508.12189/) (2025)
- [EVE: A Generator-Verifier System for Generative Policies](https://mlsanity.com/paper/2512.21430/) (2025)
- [Learning Adaptive Dexterous Grasping from Single Demonstrations](https://mlsanity.com/paper/2503.20208/) (2025)
- [Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shaping](https://mlsanity.com/paper/2511.11551/) (2025)
- [PPGuide: Steering Diffusion Policies with Performance Predictive Guidance](https://mlsanity.com/paper/2603.10980/) (2026)
- [Refining Compositional Diffusion for Reliable Long-Horizon Planning](https://mlsanity.com/paper/2605.03075/) (2026)
- [Inference-time Policy Steering via Vision and Touch](https://mlsanity.com/paper/2606.14981/) (2026)
- [MSG: Multi-Stream Generative Policies for Sample-Efficient Robotic Manipulation](https://mlsanity.com/paper/2509.24956/) (2025)
- [SketchPlan: Diffusion Based Drone Planning From Human Sketches](https://mlsanity.com/paper/2510.03545/) (2025)
- [SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLA](https://mlsanity.com/paper/2606.15568/) (2026)
- [Training-Free Action Correction for VLA Model Failures via Language Feedback](https://mlsanity.com/paper/2608.29967/) (2026)
- [Steering Generative Robot Policies with Lexicographic Preferences](https://mlsanity.com/paper/2609.15014/) (2026)
- [Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG](https://mlsanity.com/paper/2603.09011/) (2026)

## Abstract (arXiv)

Generative policies trained with human demonstrations can autonomously accomplish multimodal, long-horizon tasks. However, during inference, humans are often removed from the policy execution loop, limiting the ability to guide a pre-trained policy towards a specific sub-goal or trajectory shape among multiple predictions. Naive human intervention may inadvertently exacerbate distribution shift, leading to constraint violations or execution failures. To better align policy output with human intent without inducing out-of-distribution errors, we propose an Inference-Time Policy Steering (ITPS) framework that leverages human interactions to bias the generative sampling process, rather than fine-tuning the policy on interaction data. We evaluate ITPS across three simulated and real-world benchmarks, testing three forms of human interaction and associated alignment distance metrics. Among six sampling strategies, our proposed stochastic sampling with diffusion policy achieves the best trade-off between alignment and distribution shift. Videos are available at https://yanweiw.github.io/itps/.