# Asynchronous Temporal Fields for Action Recognition

arXiv: https://arxiv.org/abs/1612.06371 · HTML: https://mlsanity.com/paper/1612.06371/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2016.

## Built on by (most cited first, 21 of 21)

- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Video-Text Representation Learning via Differentiable Weak Temporal Alignment](https://mlsanity.com/paper/2203.16784/) (2022)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [CTRN: Class-Temporal Relational Network for Action Detection](https://mlsanity.com/paper/2110.13473/) (2021)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations](https://mlsanity.com/paper/2105.07085/) (2021)
- [Multi-dataset Training of Transformers for Robust Action Recognition](https://mlsanity.com/paper/2209.12362/) (2022)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition](https://mlsanity.com/paper/2104.01110/) (2021)
- [Distillation of Human-Object Interaction Contexts for Action Recognition](https://mlsanity.com/paper/2112.09448/) (2021)
- [No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration](https://mlsanity.com/paper/2211.00549/) (2022)
- [Spatial-Temporal Alignment Network for Action Recognition](https://mlsanity.com/paper/2308.09897/) (2023)
- [AAN: Attributes-Aware Network for Temporal Action Detection](https://mlsanity.com/paper/2309.00696/) (2023)
- [Dual-Stream Alignment for Action Segmentation](https://mlsanity.com/paper/2510.07652/) (2025)
- [Texture-Based Input Feature Selection for Action Recognition](https://mlsanity.com/paper/2303.00138/) (2023)
- [Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2308.05274/) (2023)
- [BIT: Bi-Level Temporal Modeling for Efficient Supervised Action Segmentation](https://mlsanity.com/paper/2308.14900/) (2023)
- [Self-supervised Extraction of Human Motion Structures via Frame-wise Discrete Features](https://mlsanity.com/paper/2309.05972/) (2023)

## Abstract (arXiv)

Actions are more than just movements and trajectories: we cook to eat and we hold a cup to drink from it. A thorough understanding of videos requires going beyond appearance modeling and necessitates reasoning about the sequence of activities, as well as the higher-level constructs such as intentions. But how do we model and reason about these? We propose a fully-connected temporal CRF model for reasoning over various aspects of activities that includes objects, actions, and intentions, where the potentials are predicted by a deep network. End-to-end training of such structured models is a challenging endeavor: For inference and learning we need to construct mini-batches consisting of whole videos, leading to mini-batches with only a few videos. This causes high-correlation between data points leading to breakdown of the backprop algorithm. To address this challenge, we present an asynchronous variational inference method that allows efficient end-to-end training. Our method achieves a classification mAP of 22.4% on the Charades benchmark, outperforming the state-of-the-art (17.2% mAP), and offers equal gains on the task of temporal localization.