mlsanityData to 14 Oct 2026

Multi-task Learning For Joint Action and Gesture Recognition

arXiv 2505.17867 ↗First posted 23 May 2025cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

In practical applications, computer vision tasks often need to be addressed simultaneously. Multitask learning typically achieves this by jointly training a single deep neural network to learn shared representations, providing efficiency and improving generalization. Although action and gesture recognition are closely related tasks, since they focus on body and hand movements, current state-of-the-art methods handle them separately. In this paper, we show that employing a multi-task learning paradigm for action and gesture recognition results in more efficient, robust and generalizable visual representations, by leveraging the synergies between these tasks. Extensive experiments on multiple action and gesture datasets demonstrate that handling actions and gestures in a single architecture can achieve better performance for both tasks in comparison to their single-task learning variants.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

31 of 40 references.

References · 31

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  7. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  8. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  9. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  10. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  11. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  12. 2016Cross-stitch Networks for Multi-task Learningcited by 467
19 more
  1. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  2. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  3. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  4. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  5. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  6. 2018Auxiliary Tasks in Multi-task Learningcited by 108
  7. 2016Trace Norm Regularised Deep Multi-Task Learningcited by 50
  8. 2022MulT: An End-to-End Multitask Learning Transformercited by 48
  9. 2019Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognitioncited by 35
  10. 2018Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognitioncited by 15
  11. 2021Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognitioncited by 13
  12. 2024SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Imagescited by 13
  13. 2018SUSiNet: See, Understand and Summarize itcited by 8
  14. 2021Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognitioncited by 7
  15. 2024GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognitioncited by 7
  16. 2024Enhancing Video Transformers for Action Understanding with VLM-aided Trainingcited by 6
  17. 2020Learned Weight Sharing for Deep Multi-Task Learning by Natural Evolution Strategy and Stochastic Gradient Descentcited by 2
  18. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
  19. 2023Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognitioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Neuro-Symbolic Graph Transformers for Hierarchical Intention Recognition from Multimodal Human Activity Sequencescited by 0