# FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation

arXiv: https://arxiv.org/abs/2306.11046 · HTML: https://mlsanity.com/paper/2306.11046/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 19 Jun 2023.

## Builds on (most cited first, 30 of 34)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Divergence-aware Federated Self-Supervised Learning](https://mlsanity.com/paper/2204.04385/) (2022)
- [Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2206.13028/) (2022)
- [P-CNN: Pose-based CNN Features for Action Recognition](https://mlsanity.com/paper/1506.03607/) (2015)
- [Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction](https://mlsanity.com/paper/1909.13245/) (2019)
- [SPAct: Self-supervised Privacy Preservation for Action Recognition](https://mlsanity.com/paper/2203.15205/) (2022)
- [Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation](https://mlsanity.com/paper/2203.10144/) (2022)
- [Decentralised Learning from Independent Multi-Domain Labels for Person Re-Identification](https://mlsanity.com/paper/2006.04150/) (2020)
- [Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation](https://mlsanity.com/paper/2203.06338/) (2022)
- [Federated Self-supervised Learning for Video Understanding](https://mlsanity.com/paper/2207.01975/) (2022)

## Built on by (most cited first, 5 of 5)

- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Federated Client-tailored Adapter for Medical Image Segmentation](https://mlsanity.com/paper/2504.18020/) (2025)
- [Universal Skeleton Understanding via Differentiable Rendering and MLLMs](https://mlsanity.com/paper/2603.18003/) (2026)
- [Federated Action Recognition for Smart Worker Assistance Using FastPose](https://mlsanity.com/paper/2508.14113/) (2025)
- [Privacy-Preserving Action Recognition: Taxonomy, Methods, and Privacy-Utility Trade-offs](https://mlsanity.com/paper/2608.04501/) (2026)

## Abstract (arXiv)

Existing skeleton-based action recognition methods typically follow a centralized learning paradigm, which can pose privacy concerns when exposing human-related videos. Federated Learning (FL) has attracted much attention due to its outstanding advantages in privacy-preserving. However, directly applying FL approaches to skeleton videos suffers from unstable training. In this paper, we investigate and discover that the heterogeneous human topology graph structure is the crucial factor hindering training stability. To address this limitation, we pioneer a novel Federated Skeleton-based Action Recognition (FSAR) paradigm, which enables the construction of a globally generalized model without accessing local sensitive data. Specifically, we introduce an Adaptive Topology Structure (ATS), separating generalization and personalization by learning a domain-invariant topology shared across clients and a domain-specific topology decoupled from global model aggregation.Furthermore, we explore Multi-grain Knowledge Distillation (MKD) to mitigate the discrepancy between clients and server caused by distinct updating patterns through aligning shallow block-wise motion features. Extensive experiments on multiple datasets demonstrate that FSAR outperforms state-of-the-art FL-based methods while inherently protecting privacy.