# ProVox: Personalization and Proactive Planning for Situated Human-Robot Collaboration

arXiv: https://arxiv.org/abs/2506.12248 · HTML: https://mlsanity.com/paper/2506.12248/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Shared Autonomy via Hindsight Optimization for Teleoperation and Teaming](https://mlsanity.com/paper/1706.00155/) (2017)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [Learning Preferences for Manipulation Tasks from Online Coactive Feedback](https://mlsanity.com/paper/1601.00741/) (2016)
- [Learning Adaptive Language Interfaces through Decomposition](https://mlsanity.com/paper/2010.05190/) (2020)
- [Naturalizing a Programming Language via Interactive Learning](https://mlsanity.com/paper/1704.06956/) (2017)
- [Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity](https://mlsanity.com/paper/2404.03570/) (2024)
- [Gesture-Informed Robot Assistance via Foundation Models](https://mlsanity.com/paper/2309.02721/) (2023)
- [APReL: A Library for Active Preference-based Reward Learning Algorithms](https://mlsanity.com/paper/2108.07259/) (2021)
- [MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking](https://mlsanity.com/paper/2402.18796/) (2024)
- [In-Mouth Robotic Bite Transfer with Visual and Haptic Sensing](https://mlsanity.com/paper/2211.12705/) (2022)
- [Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2411.02599/) (2024)

## Built on by (most cited first, 2 of 2)

- [Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation](https://mlsanity.com/paper/2603.02623/) (2026)
- [Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting](https://mlsanity.com/paper/2512.20014/) (2025)

## Abstract (arXiv)

Collaborative robots must quickly adapt to their partner's intent and preferences to proactively identify helpful actions. This is especially true in situated settings where human partners can continually teach robots new high-level behaviors, visual concepts, and physical skills (e.g., through demonstration), growing the robot's capabilities as the human-robot pair work together to accomplish diverse tasks. In this work, we argue that robots should be able to infer their partner's goals from early interactions and use this information to proactively plan behaviors ahead of explicit instructions from the user. Building from the strong commonsense priors and steerability of large language models, we introduce ProVox ("Proactive Voice"), a novel framework that enables robots to efficiently personalize and adapt to individual collaborators. We design a meta-prompting protocol that empowers users to communicate their distinct preferences, intent, and expected robot behaviors ahead of starting a physical interaction. ProVox then uses the personalized prompt to condition a proactive language model task planner that anticipates a user's intent from the current interaction context and robot capabilities to suggest helpful actions; in doing so, we alleviate user burden, minimizing the amount of time partners spend explicitly instructing and supervising the robot. We evaluate ProVox through user studies grounded in household manipulation tasks (e.g., assembling lunch bags) that measure the efficiency of the collaboration, as well as features such as perceived helpfulness, ease of use, and reliability. Our analysis suggests that both meta-prompting and proactivity are critical, resulting in 38.7% faster task completion times and 31.9% less user burden relative to non-active baselines. Supplementary material, code, and videos can be found at https://provox-2025.github.io.