mlsanityData to 14 Oct 2026

Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization

arXiv 2308.03950 ↗First posted 7 Aug 2023cs.CV cs.MMcited by 24

Not on the map. Why?

Abstract (arXiv)

Zero-shot skeleton-based action recognition aims to recognize actions of unseen categories after training on data of seen categories. The key is to build the connection between visual and semantic space from seen to unseen classes. Previous studies have primarily focused on encoding sequences into a singular feature vector, with subsequent mapping the features to an identical anchor point within the embedded space. Their performance is hindered by 1) the ignorance of the global visual/semantic distribution alignment, which results in a limitation to capture the true interdependence between the two spaces. 2) the negligence of temporal information since the frame-wise features with rich action clues are directly pooled into a single feature vector. We propose a new zero-shot skeleton-based action recognition method via mutual information (MI) estimation and maximization. Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed. Extensive experiments on three large-scale skeleton action datasets confirm the effectiveness of our method. Code: https://github.com/YujieOuO/SMIE.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

36 of 49 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  7. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,209
  8. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  9. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  10. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  11. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  12. 2019On Mutual Information Maximization for Representation Learningcited by 316
24 more
  1. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  2. 2019Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognitioncited by 135
  3. 2017A New Representation of Skeleton Sequences for 3D Action Recognitioncited by 126
  4. 20213D Human Action Representation Learning via Cross-View Consistency Pursuitcited by 106
  5. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  6. 2020MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognitioncited by 92
  7. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 90
  8. 2017Interpretable 3D Human Action Analysis with Temporal Convolutional Networkscited by 89
  9. 2020Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applicationscited by 80
  10. 2021Skeleton-Contrastive 3D Action Representation Learningcited by 76
  11. 2017Learning Robust Visual-Semantic Embeddingscited by 56
  12. 2018Towards Universal Representation for Unseen Action Recognitioncited by 44
  13. 2019Rethinking Zero-Shot Learning: A Conditional Visual Classification Perspectivecited by 42
  14. 2020Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognitioncited by 34
  15. 2019Locality and compositionality in zero-shot learningcited by 29
  16. 2016Zero-Shot Visual Recognition via Bidirectional Latent Embeddingcited by 27
  17. 2016Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentationcited by 27
  18. 2023Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequencescited by 25
  19. 2021Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognitioncited by 24
  20. 2019Skeleton based Zero Shot Action Recognition in Joint Pose-Language Semantic Spacecited by 19
  21. 2021Weakly-supervised 3D Human Pose Estimation with Cross-view U-shaped Graph Convolutional Networkcited by 13
  22. 2022Interventional Contrastive Learning with Meta Semantic Regularizercited by 11
  23. 2023Spatio-Temporal Branching for Motion Prediction using Motion Incrementscited by 9
  24. 2022Robust Local Preserving and Global Aligning Network for Adversarial Domain Adaptationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 6; 2026: 10.

24 papers.

Sort:

By area:

  1. 2024Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognitioncited by 23
  2. 2024Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognitioncited by 19
  3. 2024SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoderscited by 19
  4. 2024Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognitioncited by 13
  5. 2024Zero-Shot Skeleton-based Action Recognition with Dual Visual-Text Alignmentcited by 9
  6. 2024Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognitioncited by 9
  7. 2024An Information Compensation Framework for Zero-Shot Skeleton-based Action Recognitioncited by 7
  8. 2023Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Surveycited by 6
  9. 20253D Skeleton-Based Action Recognition: A Reviewcited by 6
  10. 2025Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognitioncited by 6
  11. 2025Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignmentcited by 4
  12. 2025Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognitioncited by 3
12 more
  1. 2025Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptationcited by 3
  2. 2026Universal Skeleton Understanding via Differentiable Rendering and MLLMscited by 3
  3. 2026SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognitioncited by 3
  4. 2026Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognitioncited by 2
  5. 2025DynaPURLS: Dynamic Refinement of Part-Aware Representations for Skeleton-Based Zero-Shot Action Recognitioncited by 1
  6. 2026SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognitioncited by 1
  7. 2026Toward Universal Skeleton-Based Action Recognition across Heterogeneous Skeletons and Open Vocabulariescited by 0
  8. 2026Marrying Text-to-Motion Generation with Skeleton-Based Action Recognitioncited by 0
  9. 2026PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognitioncited by 0
  10. 2026GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognitioncited by 0
  11. 2026Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognitioncited by 0
  12. 2026Zero-Shot Skeleton-Based Action Anticipationcited by 0