# Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception

arXiv: https://arxiv.org/abs/2603.13176 · HTML: https://mlsanity.com/paper/2603.13176/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders](https://mlsanity.com/paper/1510.01442/) (2015)
- [Context-Aware Streaming Perception in Dynamic Environments](https://mlsanity.com/paper/2208.07479/) (2022)
- [Enhanced Human-Robot Collaboration using Constrained Probabilistic Human-Motion Prediction](https://mlsanity.com/paper/2310.03314/) (2023)
- [Bayesian Intention for Enhanced Human Robot Collaboration](https://mlsanity.com/paper/2410.00302/) (2024)
- [VIoTGPT: Learning to Schedule Vision Tools in LLMs towards Intelligent Video Internet of Things](https://mlsanity.com/paper/2312.00401/) (2023)
- [Detecting the Starting Frame of Actions in Video](https://mlsanity.com/paper/1906.03340/) (2019)
- [How Does Perception Affect Safety: New Metrics and Strategy](https://mlsanity.com/paper/2312.07744/) (2023)
- [Relevance for Human Robot Collaboration](https://mlsanity.com/paper/2409.07753/) (2024)
- [Eye Movement Feature-Guided Signal De-Drifting in Electrooculography Systems](https://mlsanity.com/paper/2509.07416/) (2025)
- [Latency vs precision: Stability preserving perception scheduling](https://mlsanity.com/paper/2401.13585/) (2024)

## Abstract (arXiv)

In modern human-robot collaboration (HRC) applications, multiple perception modules jointly extract visual, auditory, and contextual cues to achieve comprehensive scene understanding, enabling the robot to provide appropriate assistance to human agents intelligently. While executing multiple perception modules on a frame-by-frame basis enhances perception quality in offline settings, it inevitably accumulates latency, leading to a substantial decline in system performance in streaming perception scenarios. Recent work in scene understanding, termed Relevance, has established a solid foundation for developing efficient methodologies in HRC. However, modern perception pipelines still face challenges related to information redundancy and suboptimal allocation of computational resources. Drawing inspiration from the Relevance concept and the information sparsity in HRC events, we propose a novel lightweight perception scheduling framework that efficiently leverages output from previous frames to estimate and schedule necessary perception modules in real-time based on scene context. The experimental results demonstrate that the proposed perception scheduling framework effectively reduces computational latency by up to 27.52% compared to conventional parallel perception pipelines, while also achieving a 72.73% improvement in MMPose activation recall. Additionally, the framework demonstrates high keyframe accuracy, achieving rates of up to 98%. The results validate the framework's capability to enhance real-time perception efficiency without significantly compromising accuracy. The framework shows potential as a scalable and systematic solution for multimodal streaming perception systems in HRC.