# An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos

arXiv: https://arxiv.org/abs/2410.02152 · HTML: https://mlsanity.com/paper/2410.02152/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective](https://mlsanity.com/paper/2208.07365/) (2022)
- [Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performances](https://mlsanity.com/paper/2303.10280/) (2023)
- [Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training](https://mlsanity.com/paper/2312.02914/) (2023)

## Built on by (most cited first, 1 of 1)

- [How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?](https://mlsanity.com/paper/2506.20795/) (2025)

## Abstract (arXiv)

In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.