# Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization

arXiv: https://arxiv.org/abs/2211.14053 · HTML: https://mlsanity.com/paper/2211.14053/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 79 references that are in the map.
- First posted on arXiv 25 Nov 2022.

## Builds on (most cited first, 30 of 62)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)

## Built on by (most cited first, 19 of 19)

- [EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries](https://mlsanity.com/paper/2212.06969/) (2022)
- [Large-capacity and Flexible Video Steganography via Invertible Neural Network](https://mlsanity.com/paper/2304.12300/) (2023)
- [MMAD: Multi-label Micro-Action Detection in Videos](https://mlsanity.com/paper/2407.05311/) (2024)
- [Adapting Short-Term Transformers for Action Detection in Untrimmed Videos](https://mlsanity.com/paper/2312.01897/) (2023)
- [HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization](https://mlsanity.com/paper/2408.06437/) (2024)
- [OpenTAD: A Unified Framework and Comprehensive Study of Temporal Action Detection](https://mlsanity.com/paper/2502.20361/) (2025)
- [Learning to Identify Critical States for Reinforcement Learning from Videos](https://mlsanity.com/paper/2308.07795/) (2023)
- [Temporal Action Detection Model Compression by Progressive Block Drop](https://mlsanity.com/paper/2503.16916/) (2025)
- [Online Generic Event Boundary Detection](https://mlsanity.com/paper/2510.06855/) (2025)
- [LoSA: Long-Short-range Adapter for Scaling End-to-End Temporal Action Localization](https://mlsanity.com/paper/2404.01282/) (2024)
- [MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detection](https://mlsanity.com/paper/2511.17929/) (2025)
- [Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning](https://mlsanity.com/paper/2401.04105/) (2024)
- [EvoStruggle: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levels](https://mlsanity.com/paper/2510.01362/) (2025)
- [A Hybrid Graph Network for Complex Activity Detection in Video](https://mlsanity.com/paper/2310.17493/) (2023)
- [Towards Weakly Supervised End-to-end Learning for Long-video Action Recognition](https://mlsanity.com/paper/2311.17118/) (2023)
- [Hierarchical Multi-Stage Transformer Architecture for Context-Aware Temporal Action Localization](https://mlsanity.com/paper/2507.06411/) (2025)
- [Generative Model-Based Feature Attention Module for Video Action Analysis](https://mlsanity.com/paper/2508.13565/) (2025)
- [OmniFD: A Unified Model for Versatile Face Forgery Detection](https://mlsanity.com/paper/2512.01128/) (2025)
- [Reversible Efficient Diffusion for Image Fusion](https://mlsanity.com/paper/2601.20260/) (2026)

## Abstract (arXiv)

Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content. Given limited GPU memory, training TAL end to end (i.e., from videos to predictions) on long videos is a significant challenge. Most methods can only train on pre-extracted features without optimizing them for the localization problem, consequently limiting localization performance. In this work, to extend the potential in TAL networks, we propose a novel end-to-end method Re2TAL, which rewires pretrained video backbones for reversible TAL. Re2TAL builds a backbone with reversible modules, where the input can be recovered from the output such that the bulky intermediate activations can be cleared from memory during training. Instead of designing one single type of reversible module, we propose a network rewiring mechanism, to transform any module with a residual connection to a reversible module without changing any parameters. This provides two benefits: (1) a large variety of reversible networks are easily obtained from existing and even future model designs, and (2) the reversible models require much less training effort as they reuse the pre-trained parameters of their original non-reversible versions. Re2TAL, only using the RGB modality, reaches 37.01% average mAP on ActivityNet-v1.3, a new state-of-the-art record, and mAP 64.9% at tIoU=0.5 on THUMOS-14, outperforming all other RGB-only methods.