# Relevance for Human Robot Collaboration

arXiv: https://arxiv.org/abs/2409.07753 · HTML: https://mlsanity.com/paper/2409.07753/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Deep Object-Centric Representations for Generalizable Robot Learning](https://mlsanity.com/paper/1708.04225/) (2017)
- [Visual Saliency Detection Based on Multiscale Deep CNN Features](https://mlsanity.com/paper/1609.02077/) (2016)
- [When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detection](https://mlsanity.com/paper/2202.11911/) (2022)
- [Recurrent Attentional Networks for Saliency Detection](https://mlsanity.com/paper/1604.03227/) (2016)
- [InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions](https://mlsanity.com/paper/2311.12943/) (2023)
- [Enhanced Human-Robot Collaboration using Constrained Probabilistic Human-Motion Prediction](https://mlsanity.com/paper/2310.03314/) (2023)
- [COVERED, CollabOratiVE Robot Environment Dataset for 3D Semantic segmentation](https://mlsanity.com/paper/2302.12656/) (2023)
- [How Does Perception Affect Safety: New Metrics and Strategy](https://mlsanity.com/paper/2312.07744/) (2023)

## Built on by (most cited first, 2 of 2)

- [Relevance-driven Decision Making for Safer and More Efficient Human Robot Collaboration](https://mlsanity.com/paper/2409.13998/) (2024)
- [Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception](https://mlsanity.com/paper/2603.13176/) (2026)

## Abstract (arXiv)

Inspired by the human ability to selectively focus on relevant information, this paper introduces relevance, a novel dimensionality reduction process for human-robot collaboration (HRC). Our approach incorporates a continuously operating perception module, evaluates cue sufficiency within the scene, and applies a flexible formulation and computation framework. To accurately and efficiently quantify relevance, we developed an event-based framework that maintains a continuous perception of the scene and selectively triggers relevance determination. Within this framework, we developed a probabilistic methodology, which considers various factors and is built on a novel structured scene representation. Simulation results demonstrate that the relevance framework and methodology accurately predict the relevance of a general HRC setup, achieving a precision of 0.99, a recall of 0.94, an F1 score of 0.96, and an object ratio of 0.94. Relevance can be broadly applied to several areas in HRC to accurately improve task planning time by 79.56% compared with pure planning for a cereal task, reduce perception latency by up to 26.53% for an object detector, improve HRC safety by up to 13.50% and reduce the number of inquiries for HRC by 80.84%. A real-world demonstration showcases the relevance framework's ability to intelligently and seamlessly assist humans in everyday tasks.