# Lightweight Network Architecture for Real-Time Action Recognition

arXiv: https://arxiv.org/abs/1905.08711 · HTML: https://mlsanity.com/paper/1905.08711/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2019.

## Built on by (most cited first, 10 of 10)

- [Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation](https://mlsanity.com/paper/2103.16024/) (2021)
- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)
- [VideoMamba: Spatio-Temporal Selective State Space Model](https://mlsanity.com/paper/2407.08476/) (2024)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Fine-tuning of sign language recognition models: a technical report](https://mlsanity.com/paper/2302.07693/) (2023)
- [Deep learning-based approaches for human motion decoding in smart walkers for rehabilitation](https://mlsanity.com/paper/2301.05575/) (2023)
- [Online Gesture Recognition using Transformer and Natural Language Processing](https://mlsanity.com/paper/2305.03407/) (2023)
- [A Transformer Architecture for Online Gesture Recognition of Mathematical Expressions](https://mlsanity.com/paper/2211.02643/) (2022)
- [ISLR101: an Iranian Word-Level Sign Language Recognition Dataset](https://mlsanity.com/paper/2503.12451/) (2025)

## Abstract (arXiv)

In this work we present a new efficient approach to Human Action Recognition called Video Transformer Network (VTN). It leverages the latest advances in Computer Vision and Natural Language Processing and applies them to video understanding. The proposed method allows us to create lightweight CNN models that achieve high accuracy and real-time speed using just an RGB mono camera and general purpose CPU. Furthermore, we explain how to improve accuracy by distilling from multiple models with different modalities into a single model. We conduct a comparison with state-of-the-art methods and show that our approach performs on par with most of them on famous Action Recognition datasets. We benchmark the inference time of the models using the modern inference framework and argue that our approach compares favorably with other methods in terms of speed/accuracy trade-off, running at 56 FPS on CPU. The models and the training code are available.