# Where Should I Join? Robot Group Joining via Language-Guided Goal Prediction

arXiv: https://arxiv.org/abs/2609.28467 · HTML: https://mlsanity.com/paper/2609.28467/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 27 of 27)

- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [SACSoN: Scalable Autonomous Control for Social Navigation](https://mlsanity.com/paper/2306.01874/) (2023)
- [VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models](https://mlsanity.com/paper/2404.00210/) (2024)
- [NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2506.23468/) (2025)
- [OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation](https://mlsanity.com/paper/2509.19480/) (2025)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [RTMW: Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation](https://mlsanity.com/paper/2407.08634/) (2024)
- [OLiVia-Nav: An Online Lifelong Vision Language Approach for Mobile Robot Social Navigation](https://mlsanity.com/paper/2409.13675/) (2024)
- [F-formation Detection: Individuating Free-standing Conversational Groups in Images](https://mlsanity.com/paper/1409.2702/) (2014)
- [Group-based Motion Prediction for Navigation in Crowded Environments](https://mlsanity.com/paper/2107.11637/) (2021)
- [Towards More Practical Group Activity Detection: A New Benchmark and Model](https://mlsanity.com/paper/2312.02878/) (2023)
- [ConfLab: A Data Collection Concept, Dataset, and Benchmark for Machine Analysis of Free-Standing Social Interactions in the Wild](https://mlsanity.com/paper/2205.05177/) (2022)
- [REFORM: Recognizing F-formations for Social Robots](https://mlsanity.com/paper/2008.07668/) (2020)
- [Self-supervised Social Relation Representation for Human Group Detection](https://mlsanity.com/paper/2203.03843/) (2022)
- [Let me join you! Real-time F-formation recognition by a socially aware robot](https://mlsanity.com/paper/2008.10078/) (2020)
- [From Attributes to Natural Language: A Survey and Foresight on Text-based Person Re-identification](https://mlsanity.com/paper/2408.00096/) (2024)
- [From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection](https://mlsanity.com/paper/2602.09002/) (2026)
- [Real-time Trajectory-based Social Group Detection](https://mlsanity.com/paper/2304.05678/) (2023)
- [Dynamic Group Detection using VLM-augmented Temporal Groupness Graph](https://mlsanity.com/paper/2509.04758/) (2025)
- [Detecting socially interacting groups using f-formation: A survey of taxonomy, methods, datasets, applications, challenges, and future research directions](https://mlsanity.com/paper/2108.06181/) (2021)
- [Learning Social Cost Functions for Human-Aware Path Planning](https://mlsanity.com/paper/2407.10547/) (2024)

## Abstract (arXiv)

Social navigation typically assumes a specified goal and focuses on reaching it while respecting social conventions, whereas robot group joining requires predicting where to join based on the group's real-time activity and formation. This is a highly semantic task, yet an important capability for applications such as robotic guide dogs and autonomous mobility scooters. We formulate language-grounded robot group joining: given an observation and a natural-language description of a target group, the robot identifies the relevant group members and predicts socially compliant joining poses. For grounding, we generate structured candidate subsets through recursive spectral partitioning and rank them with a language-conditioned image--geometry model. Given the grounded group, a goal predictor leverages human-formation priors to produce a multimodal energy--orientation map over feasible robot poses. Experiments on conversations, queues, and audiences across varying group sizes, crowd densities, and visual ambiguities show that our method achieves competitive grounding accuracy with sub-second inference and outperforms all baselines in joining-pose prediction. Real-robot experiments further demonstrate group joining in both static and dynamically changing interactions.