# Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration

arXiv: https://arxiv.org/abs/2512.04453 · HTML: https://mlsanity.com/paper/2512.04453/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 70 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [PDDLGym: Gym Environments from PDDL Problems](https://mlsanity.com/paper/2002.06432/) (2020)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity](https://mlsanity.com/paper/2402.06529/) (2024)
- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)
- [Inferring the Goals of Communicating Agents from Actions and Instructions](https://mlsanity.com/paper/2306.16207/) (2023)
- [Learning to Assist Humans without Inferring Rewards](https://mlsanity.com/paper/2411.02623/) (2024)
- [Intrinsically motivated reinforcement learning for human-robot interaction in the real-world](https://mlsanity.com/paper/1804.05259/) (2018)
- [I Know What You Meant: Learning Human Objectives by (Under)estimating Their Choice Set](https://mlsanity.com/paper/2011.06118/) (2020)
- [Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racing](https://mlsanity.com/paper/2410.10062/) (2024)
- [COOPERA: Continual Open-Ended Human-Robot Assistance](https://mlsanity.com/paper/2510.23495/) (2025)
- [Infinite Ends from Finite Samples: Open-Ended Goal Inference as Top-Down Bayesian Filtering of Bottom-Up Proposals](https://mlsanity.com/paper/2407.16770/) (2024)
- [Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner](https://mlsanity.com/paper/1804.06819/) (2018)
- [Socially Guided Intrinsic Motivation for Robot Learning of Motor Skills](https://mlsanity.com/paper/1804.07269/) (2018)
- [Exploring the Cost of Interruptions in Human-Robot Teaming](https://mlsanity.com/paper/2311.00785/) (2023)
- [Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2410.13957/) (2024)
- [I've Changed My Mind: Robots Adapting to Changing Human Goals during Collaboration](https://mlsanity.com/paper/2511.15914/) (2025)
- [Improved Inference of Human Intent by Combining Plan Recognition and Language Feedback](https://mlsanity.com/paper/2310.02462/) (2023)
- [The intrinsic motivation of reinforcement and imitation learning for sequential tasks](https://mlsanity.com/paper/2412.20573/) (2024)
- [A Motivational Architecture for Open-Ended Learning Challenges in Robots](https://mlsanity.com/paper/2506.18454/) (2025)
- [Maximal Adaptation, Minimal Guidance: Permissive Reactive Robot Task Planning with Humans in the Loop](https://mlsanity.com/paper/2510.12662/) (2025)

## Built on by (most cited first, 5 of 5)

- ["I Didn't Make the Micro Decisions": Measuring, Inducing, and Exposing Goal-Level AI Contributions in Collaboration](https://mlsanity.com/paper/2605.21363/) (2026)
- [Robots Influencing Humans to Reveal their Goals during Collaboration and Competition](https://mlsanity.com/paper/2609.05519/) (2026)
- [Learning Transferable Latent User Preferences for Human-Aligned Decision Making](https://mlsanity.com/paper/2605.12682/) (2026)
- [Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration](https://mlsanity.com/paper/2609.13982/) (2026)
- [HelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution](https://mlsanity.com/paper/2610.17694/) (2026)

## Abstract (arXiv)

To collaborate with humans, robots must infer goals that are often ambiguous, difficult to articulate, or not drawn from a fixed set. Prior approaches restrict inference to a predefined goal set, rely only on observed actions, or depend exclusively on explicit instructions, making them brittle in real-world interactions. We present BALI (Bidirectional Action-Language Inference) for goal prediction, a method that integrates natural language preferences with observed human actions in a receding-horizon planning tree. BALI combines language and action cues from the human, asks clarifying questions only when the expected information gain from the answer outweighs the cost of interruption, and selects supportive actions that align with inferred goals. We evaluate the approach in collaborative cooking tasks, where goals may be novel to the robot and unbounded. Compared to baselines, BALI yields more stable goal predictions and significantly fewer mistakes.