# Temporal Zoom Networks: Distance Regression and Continuous Depth for Efficient Action Localization

arXiv: https://arxiv.org/abs/2511.03943 · HTML: https://mlsanity.com/paper/2511.03943/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 43 references that are in the map.
- First posted on arXiv 6 Nov 2025.

## Builds on (most cited first, 30 of 34)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Boundary loss for highly unbalanced segmentation](https://mlsanity.com/paper/1812.07032/) (2018)
- [Dynamic Head: Unifying Object Detection Heads with Attentions](https://mlsanity.com/paper/2106.08322/) (2021)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [Fisher-Rao Metric, Geometry, and Complexity of Neural Networks](https://mlsanity.com/paper/1711.01530/) (2017)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)

## Abstract (arXiv)

Temporal action localization requires both precise boundary detection and computational efficiency. Current methods apply uniform computation across all temporal positions, wasting resources on easy boundaries while struggling with ambiguous ones. We address this through two complementary innovations: Boundary Distance Regression (BDR), which replaces classification-based boundary detection with signed-distance regression achieving 3.3--16.7× lower variance; and Adaptive Temporal Refinement (ATR), which allocates transformer depth continuously (τ∈[0,1]) to concentrate computation near difficult boundaries. On THUMOS14, our method achieves 56.5% mAP@0.7 and 58.2% average mAP@[0.3:0.7] with 151G FLOPs, using 36% fewer FLOPs than ActionFormer++ (55.7% mAP@0.7 at 235G). Compared to uniform baselines, we achieve +2.9% mAP@0.7 (+1.8% avg mAP, 5.4% relative) with 24% fewer FLOPs and 29% lower latency, with particularly strong gains on short actions (+4.2%, 8.6% relative). Training requires 1.29× baseline FLOPs, but this one-time cost is amortized over many inference runs; knowledge distillation further reduces this to 1.1× while retaining 99.5% accuracy. Our contributions include: (i) a theoretically-grounded distance formulation with information-theoretic analysis showing optimal variance scaling; (ii) a continuous depth allocation mechanism avoiding discrete routing complexity; and (iii) consistent improvements across four datasets with gains correlating with boundary heterogeneity.