# Approximated Bilinear Modules for Temporal Modeling

arXiv: https://arxiv.org/abs/2007.12887 · HTML: https://mlsanity.com/paper/2007.12887/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2020.

## Built on by (most cited first, 8 of 8)

- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Attention in Attention: Modeling Context Correlation for Efficient Video Classification](https://mlsanity.com/paper/2204.09303/) (2022)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [Optimization Planning for 3D ConvNets](https://mlsanity.com/paper/2201.04021/) (2022)
- [Boosting Video Representation Learning with Multi-Faceted Integration](https://mlsanity.com/paper/2201.04023/) (2022)
- [Video Is Graph: Structured Graph Module for Video Action Recognition](https://mlsanity.com/paper/2110.05904/) (2021)
- [Adaptive Recursive Circle Framework for Fine-grained Action Recognition](https://mlsanity.com/paper/2107.11813/) (2021)
- [ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition](https://mlsanity.com/paper/2110.15574/) (2021)

## Abstract (arXiv)

We consider two less-emphasized temporal properties of video: 1. Temporal cues are fine-grained; 2. Temporal modeling needs reasoning. To tackle both problems at once, we exploit approximated bilinear modules (ABMs) for temporal modeling. There are two main points making the modules effective: two-layer MLPs can be seen as a constraint approximation of bilinear operations, thus can be used to construct deep ABMs in existing CNNs while reusing pretrained parameters; frame features can be divided into static and dynamic parts because of visual repetition in adjacent frames, which enables temporal modeling to be more efficient. Multiple ABM variants and implementations are investigated, from high performance to high efficiency. Specifically, we show how two-layer subnets in CNNs can be converted to temporal bilinear modules by adding an auxiliary-branch. Besides, we introduce snippet sampling and shifting inference to boost sparse-frame video classification performance. Extensive ablation studies are conducted to show the effectiveness of proposed techniques. Our models can outperform most state-of-the-art methods on Something-Something v1 and v2 datasets without Kinetics pretraining, and are also competitive on other YouTube-like action recognition datasets. Our code is available on https://github.com/zhuxinqimac/abm-pytorch.