# Exploring State Change Capture of Heterogeneous Backbones @ Ego4D Hands and Objects Challenge 2022

arXiv: https://arxiv.org/abs/2211.08728 · HTML: https://mlsanity.com/paper/2211.08728/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [An Empirical Study of End-to-End Temporal Action Detection](https://mlsanity.com/paper/2204.02932/) (2022)
- [DCAN: Improving Temporal Action Detection via Dual Context Aggregation](https://mlsanity.com/paper/2112.03612/) (2021)
- [Dynamic Sampling Networks for Efficient Action Recognition in Videos](https://mlsanity.com/paper/2006.15560/) (2020)
- [BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection](https://mlsanity.com/paper/2205.02717/) (2022)

## Abstract (arXiv)

Capturing the state changes of interacting objects is a key technology for understanding human-object interactions. This technical report describes our method using heterogeneous backbones for the Ego4D Object State Change Classification and PNR Temporal Localization Challenge. In the challenge, we used the heterogeneous video understanding backbones, namely CSN with 3D convolution as operator and VideoMAE with Transformer as operator. Our method achieves an accuracy of 0.796 on OSCC while achieving an absolute temporal localization error of 0.516 on PNR. These excellent results rank 1st on the leaderboard of Ego4D OSCC & PNR-TL Challenge 2022.