# PyTorchVideo: A Deep Learning Library for Video Understanding

arXiv: https://arxiv.org/abs/2111.09887 · HTML: https://mlsanity.com/paper/2111.09887/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing](https://mlsanity.com/paper/1907.04433/) (2019)
- [A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos](https://mlsanity.com/paper/1802.08722/) (2018)

## Built on by (most cited first, 30 of 32)

- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [AugLy: Data Augmentations for Robustness](https://mlsanity.com/paper/2201.06494/) (2022)
- [WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series](https://mlsanity.com/paper/2203.09978/) (2022)
- [Words are all you need? Language as an approximation for human similarity judgments](https://mlsanity.com/paper/2206.04105/) (2022)
- [CVNets: High Performance Library for Computer Vision](https://mlsanity.com/paper/2206.02002/) (2022)
- [Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes](https://mlsanity.com/paper/2311.17948/) (2023)
- [Differentially Private Video Activity Recognition](https://mlsanity.com/paper/2306.15742/) (2023)
- [Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification](https://mlsanity.com/paper/2411.16718/) (2024)
- [A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning](https://mlsanity.com/paper/2202.13403/) (2022)
- [Know your sensORs -- A Modality Study For Surgical Action Classification](https://mlsanity.com/paper/2203.08674/) (2022)
- [VR.net: A Real-world Dataset for Virtual Reality Motion Sickness Research](https://mlsanity.com/paper/2306.03381/) (2023)
- [Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild](https://mlsanity.com/paper/2211.00794/) (2022)
- [Video RWKV:Video Action Recognition Based RWKV](https://mlsanity.com/paper/2411.05636/) (2024)
- [Long-VMNet: Accelerating Long-Form Video Understanding via Fixed Memory](https://mlsanity.com/paper/2503.13707/) (2025)
- [MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training](https://mlsanity.com/paper/2504.09844/) (2025)
- [Model-agnostic Multi-Domain Learning with Domain-Specific Adapters for Action Recognition](https://mlsanity.com/paper/2204.07270/) (2022)
- [Performance Evaluation of Action Recognition Models on Low Quality Videos](https://mlsanity.com/paper/2204.09166/) (2022)
- [VOCALExplore: Pay-as-You-Go Video Data Exploration and Model Building [Technical Report]](https://mlsanity.com/paper/2303.04068/) (2023)
- [Deep Video Codec Control for Vision Models](https://mlsanity.com/paper/2308.16215/) (2023)
- [Synthetic Human Action Video Data Generation with Pose Transfer](https://mlsanity.com/paper/2506.09411/) (2025)
- [Signal Processing in the Retina: Interpretable Graph Classifier to Predict Ganglion Cell Responses](https://mlsanity.com/paper/2401.01813/) (2024)
- [REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild](https://mlsanity.com/paper/2403.01229/) (2024)
- [Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language](https://mlsanity.com/paper/2505.13784/) (2025)
- [The Importance of Facial Features in Vision-based Sign Language Recognition: Eyes, Mouth or Full Face?](https://mlsanity.com/paper/2507.20884/) (2025)
- [Video-based Exercise Classification and Activated Muscle Group Prediction with Hybrid X3D-SlowFast Network](https://mlsanity.com/paper/2406.06703/) (2024)
- [Towards Clinical Practice in CT-Based Pulmonary Disease Screening: An Efficient and Reliable Framework](https://mlsanity.com/paper/2412.01525/) (2024)
- [Domain Generalization for Improved Human Activity Recognition in Office Space Videos Using Adaptive Pre-processing](https://mlsanity.com/paper/2503.12678/) (2025)
- [Spatiotemporal Analysis of Forest Machine Operations Using 3D Video Classification](https://mlsanity.com/paper/2505.24375/) (2025)

## Abstract (arXiv)

We introduce PyTorchVideo, an open-source deep-learning library that provides a rich set of modular, efficient, and reproducible components for a variety of video understanding tasks, including classification, detection, self-supervised learning, and low-level processing. The library covers a full stack of video understanding tools including multimodal data loading, transformations, and models that reproduce state-of-the-art performance. PyTorchVideo further supports hardware acceleration that enables real-time inference on mobile devices. The library is based on PyTorch and can be used by any training framework; for example, PyTorchLightning, PySlowFast, or Classy Vision. PyTorchVideo is available at https://pytorchvideo.org/