# Patchwork: A Patch-wise Attention Network for Efficient Object Detection and Segmentation in Video Streams

arXiv: https://arxiv.org/abs/1904.01784 · HTML: https://mlsanity.com/paper/1904.01784/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2019.

## Built on by (most cited first, 13 of 13)

- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild](https://mlsanity.com/paper/2103.10391/) (2021)
- [Active Visual Exploration Based on Attention-Map Entropy](https://mlsanity.com/paper/2303.06457/) (2023)
- [SALISA: Saliency-based Input Sampling for Efficient Video Object Detection](https://mlsanity.com/paper/2204.02397/) (2022)
- [Boggart: Towards General-Purpose Acceleration of Retrospective Video Analytics](https://mlsanity.com/paper/2106.15315/) (2021)
- [Delta Distillation for Efficient Video Processing](https://mlsanity.com/paper/2203.09594/) (2022)
- [Glimpse-Attend-and-Explore: Self-Attention for Active Visual Exploration](https://mlsanity.com/paper/2108.11717/) (2021)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale](https://mlsanity.com/paper/2404.03482/) (2024)
- [Learning to Incorporate Texture Saliency Adaptive Attention to Image Cartoonization](https://mlsanity.com/paper/2208.01587/) (2022)
- [TRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision](https://mlsanity.com/paper/2406.17483/) (2024)
- [Representation Recycling for Streaming Video Analysis](https://mlsanity.com/paper/2204.13492/) (2022)
- [A Survey on Efficiency Optimization Techniques for DNN-based Video Analytics: Process Systems, Algorithms, and Applications](https://mlsanity.com/paper/2507.15628/) (2025)

## Abstract (arXiv)

Recent advances in single-frame object detection and segmentation techniques have motivated a wide range of works to extend these methods to process video streams. In this paper, we explore the idea of hard attention aimed for latency-sensitive applications. Instead of reasoning about every frame separately, our method selects and only processes a small sub-window of the frame. Our technique then makes predictions for the full frame based on the sub-windows from previous frames and the update from the current sub-window. The latency reduction by this hard attention mechanism comes at the cost of degraded accuracy. We made two contributions to address this. First, we propose a specialized memory cell that recovers lost context when processing sub-windows. Secondly, we adopt a Q-learning-based policy training strategy that enables our approach to intelligently select the sub-windows such that the staleness in the memory hurts the performance the least. Our experiments suggest that our approach reduces the latency by approximately four times without significantly sacrificing the accuracy on the ImageNet VID video object detection dataset and the DAVIS video object segmentation dataset. We further demonstrate that we can reinvest the saved computation into other parts of the network, and thus resulting in an accuracy increase at a comparable computational cost as the original system and beating other recently proposed state-of-the-art methods in the low latency range.