# Temporal Preservation over Processing: Diagnosing and Designing Spatiotemporal Single-Stage Video Detectors

arXiv: https://arxiv.org/abs/2606.31421 · HTML: https://mlsanity.com/paper/2606.31421/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 30 of 33)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://mlsanity.com/paper/1911.11929/) (2019)
- [TAM: Temporal Adaptive Module for Video Recognition](https://mlsanity.com/paper/2005.06803/) (2020)
- [You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization](https://mlsanity.com/paper/1911.06644/) (2019)
- [Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition](https://mlsanity.com/paper/2010.11757/) (2020)
- [Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2309.07911/) (2023)
- [Linear Complexity Randomized Self-attention Mechanism](https://mlsanity.com/paper/2204.04667/) (2022)
- [Efficient Attention via Control Variates](https://mlsanity.com/paper/2302.04542/) (2023)
- [ACDnet: An action detection network for real-time edge computing based on flow-guided feature approximation and memory aggregation](https://mlsanity.com/paper/2102.13493/) (2021)
- [YOWOv2: A Stronger yet Efficient Multi-level Detection Framework for Real-time Spatio-temporal Action Detection](https://mlsanity.com/paper/2302.06848/) (2023)
- [Human-Centered Evaluation of XAI Methods](https://mlsanity.com/paper/2310.07534/) (2023)

## Abstract (arXiv)

Single-stage video object detectors are increasingly deployed in time-critical applications, yet it remains unclear whether these models genuinely reason over temporal context or merely exploit a single informative frame-a gap hidden by standard metrics, which reward correct predictions regardless of how they are reached. We address this from two complementary directions: first, we propose TemporalLens, a model-agnostic diagnostic framework probing temporal dependence through controlled perturbations, structured occlusions, temporal shuffling, redundancy injection, and resolution degradation, revealing whether a detector actually uses information across time. Applied to stacked-frame 2D detectors and our YOLO-3D architecture, it exposes behavioural differences invisible to mAP: stacked 2D models collapse when the target frame is removed, while spatiotemporal models recover predictions from earlier frames, a signature of real temporal reliance. Second, we detail YOLO-3D, a modular real-time spatiotemporal detector built on YOLOv8, and show that simply preserving temporal depth through the backbone is the dominant performance driver (+3.7 pp mAP@50 at 32 frames averaged across scales). Together, the diagnostics and architecture turn "does this detector reason over time?" into a measurable, actionable question.