# Multi-level Adversarial Spatio-temporal Learning for Footstep Pressure based FoG Detection

arXiv: https://arxiv.org/abs/2209.10770 · HTML: https://mlsanity.com/paper/2209.10770/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Enriching Local and Global Contexts for Temporal Action Localization](https://mlsanity.com/paper/2107.12960/) (2021)
- [Attentional Pooling for Action Recognition](https://mlsanity.com/paper/1711.01467/) (2017)
- [Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion](https://mlsanity.com/paper/2006.08159/) (2020)
- [Spatial-Temporal Recurrent Neural Network for Emotion Recognition](https://mlsanity.com/paper/1705.04515/) (2017)
- [Privacy-Preserving Video Classification with Convolutional Neural Networks](https://mlsanity.com/paper/2102.03513/) (2021)
- [Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks](https://mlsanity.com/paper/2103.15449/) (2021)
- [Robust Gait Recognition by Integrating Inertial and RGBD Sensors](https://mlsanity.com/paper/1610.09816/) (2016)
- [Diversifying Inference Path Selection: Moving-Mobile-Network for Landmark Recognition](https://mlsanity.com/paper/1912.00418/) (2019)

## Built on by (most cited first, 1 of 1)

- [Radio Frequency Signal based Human Silhouette Segmentation: A Sequential Diffusion Approach](https://mlsanity.com/paper/2407.19244/) (2024)

## Abstract (arXiv)

Freezing of gait (FoG) is one of the most common symptoms of Parkinson's disease, which is a neurodegenerative disorder of the central nervous system impacting millions of people around the world. To address the pressing need to improve the quality of treatment for FoG, devising a computer-aided detection and quantification tool for FoG has been increasingly important. As a non-invasive technique for collecting motion patterns, the footstep pressure sequences obtained from pressure sensitive gait mats provide a great opportunity for evaluating FoG in the clinic and potentially in the home environment. In this study, FoG detection is formulated as a sequential modelling task and a novel deep learning architecture, namely Adversarial Spatio-temporal Network (ASTN), is proposed to learn FoG patterns across multiple levels. A novel adversarial training scheme is introduced with a multi-level subject discriminator to obtain subject-independent FoG representations, which helps to reduce the over-fitting risk due to the high inter-subject variance. As a result, robust FoG detection can be achieved for unseen subjects. The proposed scheme also sheds light on improving subject-level clinical studies from other scenarios as it can be integrated with many existing deep architectures. To the best of our knowledge, this is one of the first studies of footstep pressure-based FoG detection and the approach of utilizing ASTN is the first deep neural network architecture in pursuit of subject-independent representations. Experimental results on 393 trials collected from 21 subjects demonstrate encouraging performance of the proposed ASTN for FoG detection with an AUC 0.85.