# Deep Temporal Linear Encoding Networks

arXiv: https://arxiv.org/abs/1611.06678 · HTML: https://mlsanity.com/paper/1611.06678/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2016.

## Built on by (most cited first, 27 of 27)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Stand-Alone Inter-Frame Attention in Video Models](https://mlsanity.com/paper/2206.06931/) (2022)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [Representing Videos as Discriminative Sub-graphs for Action Recognition](https://mlsanity.com/paper/2201.04027/) (2022)
- [Deep Two-Stream Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/2110.11680/) (2021)
- [Dynamic Temporal Filtering in Video Models](https://mlsanity.com/paper/2211.08252/) (2022)
- [Condensing a Sequence to One Informative Frame for Video Recognition](https://mlsanity.com/paper/2201.04022/) (2022)
- [ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation](https://mlsanity.com/paper/2203.08942/) (2022)
- [Interpretable Deep Feature Propagation for Early Action Recognition](https://mlsanity.com/paper/2107.05122/) (2021)
- [Boosting Video Representation Learning with Multi-Faceted Integration](https://mlsanity.com/paper/2201.04023/) (2022)
- [Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips](https://mlsanity.com/paper/2112.01038/) (2021)
- [Bi-Calibration Networks for Weakly-Supervised Video Representation Learning](https://mlsanity.com/paper/2206.10491/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Temporal-Distributed Backdoor Attack Against Video Based Action Recognition](https://mlsanity.com/paper/2308.11070/) (2023)
- [DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding](https://mlsanity.com/paper/2507.15569/) (2025)
- [Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling](https://mlsanity.com/paper/2210.03941/) (2022)
- [Selective Volume Mixup for Video Action Recognition](https://mlsanity.com/paper/2309.09534/) (2023)
- [A Survey on Backbones for Deep Video Action Recognition](https://mlsanity.com/paper/2405.05584/) (2024)
- [Deep Learning Computer Vision Algorithms for Real-time UAVs On-board Camera Image Processing](https://mlsanity.com/paper/2211.01037/) (2022)
- [Beyond still images: Temporal features and input variance resilience](https://mlsanity.com/paper/2311.00800/) (2023)
- [ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition](https://mlsanity.com/paper/2110.15574/) (2021)
- [Adversarial Memory Networks for Action Prediction](https://mlsanity.com/paper/2112.09875/) (2021)
- [PatchBlender: A Motion Prior for Video Transformers](https://mlsanity.com/paper/2211.14449/) (2022)
- [Texture-Based Input Feature Selection for Action Recognition](https://mlsanity.com/paper/2303.00138/) (2023)
- [QuasiNet: a neural network with trainable product layers](https://mlsanity.com/paper/2401.06137/) (2023)
- [Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis](https://mlsanity.com/paper/2502.07277/) (2025)

## Abstract (arXiv)

The CNN-encoding of features from entire videos for the representation of human actions has rarely been addressed. Instead, CNN work has focused on approaches to fuse spatial and temporal networks, but these were typically limited to processing shorter sequences. We present a new video representation, called temporal linear encoding (TLE) and embedded inside of CNNs as a new layer, which captures the appearance and motion throughout entire videos. It encodes this aggregated information into a robust video feature representation, via end-to-end learning. Advantages of TLEs are: (a) they encode the entire video into a compact feature representation, learning the semantics and a discriminative feature space; (b) they are applicable to all kinds of networks like 2D and 3D CNNs for video classification; and (c) they model feature interactions in a more expressive way and without loss of information. We conduct experiments on two challenging human action datasets: HMDB51 and UCF101. The experiments show that TLE outperforms current state-of-the-art methods on both datasets.