# Context-Aware Streaming Perception in Dynamic Environments

arXiv: https://arxiv.org/abs/2208.07479 · HTML: https://mlsanity.com/paper/2208.07479/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [MOT20: A benchmark for multi object tracking in crowded scenes](https://mlsanity.com/paper/2003.09003/) (2020)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [Improving Neural Network Quantization without Retraining using Outlier Channel Splitting](https://mlsanity.com/paper/1901.09504/) (2019)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles](https://mlsanity.com/paper/2104.07830/) (2021)
- [Deep Neural Network Compression with Single and Multiple Level Quantization](https://mlsanity.com/paper/1803.03289/) (2018)
- [Tracking the Trackers: An Analysis of the State of the Art in Multiple Object Tracking](https://mlsanity.com/paper/1704.02781/) (2017)
- [Fast Video Classification via Adaptive Cascading of Deep Models](https://mlsanity.com/paper/1611.06453/) (2016)
- [HyperSTAR: Task-Aware Hyperparameters for Deep Networks](https://mlsanity.com/paper/2005.10524/) (2020)
- [Real-Time Segmentation Networks should be Latency Aware](https://mlsanity.com/paper/2004.02574/) (2020)

## Built on by (most cited first, 8 of 8)

- [PVT++: A Simple End-to-End Latency-Aware Visual Tracking Framework](https://mlsanity.com/paper/2211.11629/) (2022)
- [Learned Two-Plane Perspective Prior based Image Resampling for Efficient Object Detection](https://mlsanity.com/paper/2303.14311/) (2023)
- [DyRoNet: Dynamic Routing and Low-Rank Adapters for Autonomous Driving Streaming Perception](https://mlsanity.com/paper/2403.05050/) (2024)
- [Real-time Stereo-based 3D Object Detection for Streaming Perception](https://mlsanity.com/paper/2410.12394/) (2024)
- [4DSegStreamer: Streaming 4D Panoptic Segmentation via Dual Threads](https://mlsanity.com/paper/2510.17664/) (2025)
- [Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception](https://mlsanity.com/paper/2603.13176/) (2026)
- [On Exploring Input Resolution Scaling For Anytime LiDAR Object Detection](https://mlsanity.com/paper/2607.08391/) (2026)
- [Understanding Dynamic Scenes at Gigapixel Scale: Wide-Area Spatio-Temporal Perception from UAVs](https://mlsanity.com/paper/2609.18210/) (2026)

## Abstract (arXiv)

Efficient vision works maximize accuracy under a latency budget. These works evaluate accuracy offline, one image at a time. However, real-time vision applications like autonomous driving operate in streaming settings, where ground truth changes between inference start and finish. This results in a significant accuracy drop. Therefore, a recent work proposed to maximize accuracy in streaming settings on average. In this paper, we propose to maximize streaming accuracy for every environment context. We posit that scenario difficulty influences the initial (offline) accuracy difference, while obstacle displacement in the scene affects the subsequent accuracy degradation. Our method, Octopus, uses these scenario properties to select configurations that maximize streaming accuracy at test time. Our method improves tracking performance (S-MOTA) by 7.4% over the conventional static approach. Further, performance improvement using our method comes in addition to, and not instead of, advances in offline accuracy.