# RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics

arXiv: https://arxiv.org/abs/2308.15368 · HTML: https://mlsanity.com/paper/2308.15368/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 93 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision](https://mlsanity.com/paper/1810.10090/) (2018)
- [Compressing Neural Networks using the Variational Information Bottleneck](https://mlsanity.com/paper/1802.10399/) (2018)
- [NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Models](https://mlsanity.com/paper/2203.15859/) (2022)
- [Robust Multi-Agent Reinforcement Learning with State Uncertainty](https://mlsanity.com/paper/2307.16212/) (2023)
- [Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognition](https://mlsanity.com/paper/2303.12512/) (2023)
- [Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles](https://mlsanity.com/paper/2104.07830/) (2021)
- [Revisiting Self-Training for Few-Shot Learning of Language Model](https://mlsanity.com/paper/2110.01256/) (2021)
- [Multi-Task Zipping via Layer-wise Neuron Sharing](https://mlsanity.com/paper/1805.09791/) (2018)
- [Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework](https://mlsanity.com/paper/2210.16798/) (2022)
- [DeepPerform: An Efficient Approach for Performance Testing of Resource-Constrained Neural Networks](https://mlsanity.com/paper/2210.05370/) (2022)
- [SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning](https://mlsanity.com/paper/2111.14693/) (2021)
- [DPMPC-Planner: A real-time UAV trajectory planning framework for complex static environments with dynamic obstacles](https://mlsanity.com/paper/2109.07024/) (2021)
- [XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse](https://mlsanity.com/paper/2211.08675/) (2022)
- [White-Box Multi-Objective Adversarial Attack on Dialogue Generation](https://mlsanity.com/paper/2305.03655/) (2023)
- [Dynamic Transformers Provide a False Sense of Efficiency](https://mlsanity.com/paper/2305.12228/) (2023)
- [NVRadarNet: Real-Time Radar Obstacle and Free Space Detection for Autonomous Driving](https://mlsanity.com/paper/2209.14499/) (2022)
- [Learning to Reverse DNNs from AI Programs Automatically](https://mlsanity.com/paper/2205.10364/) (2022)
- [Unifying and Merging Well-trained Deep Neural Networks for Inference Stage](https://mlsanity.com/paper/1805.04980/) (2018)

## Built on by (most cited first, 6 of 6)

- [DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model](https://mlsanity.com/paper/2404.05182/) (2024)
- [Real-Time Human Action Recognition on Embedded Platforms](https://mlsanity.com/paper/2409.05662/) (2024)
- [VALO: A Versatile Anytime Framework for LiDAR-based Object Detection Deep Neural Networks](https://mlsanity.com/paper/2409.11542/) (2024)
- [Visual SLAM with 3D Gaussian Primitives and Depth Priors Enabling Novel View Synthesis](https://mlsanity.com/paper/2408.05635/) (2024)
- [UniLCD: Unified Local-Cloud Decision-Making via Reinforcement Learning](https://mlsanity.com/paper/2409.11403/) (2024)
- [TetraRL: A Self-Adaptive Runtime for On-Device Deep Reinforcement Learning Systems](https://mlsanity.com/paper/2606.13891/) (2026)

## Abstract (arXiv)

Intelligent robots are designed to effectively navigate dynamic and unpredictable environments laden with moving mechanical elements and objects. Such environment-induced dynamics, including moving obstacles, can readily alter the computational demand (e.g., the creation of new tasks) and the structure of workloads (e.g., precedence constraints among tasks) during runtime, thereby adversely affecting overall system performance. This challenge is amplified when multi-task inference is expected on robots operating under stringent resource and real-time constraints. To address such a challenge, we introduce RED, a systematic real-time scheduling approach designed to support multi-task deep neural network workloads in resource-limited robotic systems. It is designed to adaptively manage the Robotic Environmental Dynamics (RED) while adhering to real-time constraints. At the core of RED lies a deadline-based scheduler that employs an intermediate deadline assignment policy, effectively managing to change workloads and asynchronous inference prompted by complex, unpredictable environments. This scheduling framework also facilitates the flexible deployment of MIMONet (multi-input multi-output neural networks), which are commonly utilized in multi-tasking robotic systems to circumvent memory bottlenecks. Building on this scheduling framework, RED recognizes and leverages a unique characteristic of MIMONet: its weight-shared architecture. To further accommodate and exploit this feature, RED devises a novel and effective workload refinement and reconstruction process. This process ensures the scheduling framework's compatibility with MIMONet and maximizes efficiency.