# A Personalized Video-Based Hand Taxonomy: Application for Individuals with Spinal Cord Injury

arXiv: https://arxiv.org/abs/2403.18094 · HTML: https://mlsanity.com/paper/2403.18094/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 25 of 25)

- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Understanding Self-supervised Learning with Dual Deep Networks](https://mlsanity.com/paper/2010.00578/) (2020)
- [Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective](https://mlsanity.com/paper/2203.08124/) (2022)
- [Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree](https://mlsanity.com/paper/1509.08985/) (2015)
- [Discriminatively Boosted Image Clustering with Fully Convolutional Auto-Encoders](https://mlsanity.com/paper/1703.07980/) (2017)
- [Differentiable Deep Clustering with Cluster Size Constraints](https://mlsanity.com/paper/1910.09036/) (2019)
- [Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study](https://mlsanity.com/paper/2002.01810/) (2020)
- [CNN features are also great at unsupervised classification](https://mlsanity.com/paper/1707.01700/) (2017)
- [Multi-Modal Deep Clustering: Unsupervised Partitioning of Images](https://mlsanity.com/paper/1912.02678/) (2019)
- [An Effective and Efficient Method for Detecting Hands in Egocentric Videos for Rehabilitation Applications](https://mlsanity.com/paper/1908.10406/) (2019)
- [Combining pretrained CNN feature extractors to enhance clustering of complex natural images](https://mlsanity.com/paper/2101.02767/) (2021)
- [Egocentric Video: A New Tool for Capturing Hand Use of Individuals with Spinal Cord Injury at Home](https://mlsanity.com/paper/1809.00928/) (2018)
- [Characterizing and comparing external measures for the assessment of cluster analysis and community detection](https://mlsanity.com/paper/2102.00708/) (2021)
- [A survey on feature weighting based K-Means algorithms](https://mlsanity.com/paper/1601.03483/) (2015)
- [SA-Net: A deep spectral analysis network for image clustering](https://mlsanity.com/paper/2009.07026/) (2020)
- [Measuring hand use in the home after cervical spinal cord injury using egocentric video](https://mlsanity.com/paper/2203.16996/) (2022)

## Abstract (arXiv)

Hand function is critical for our interactions and quality of life. Spinal cord injuries (SCI) can impair hand function, reducing independence. A comprehensive evaluation of function in home and community settings requires a hand grasp taxonomy for individuals with impaired hand function. Developing such a taxonomy is challenging due to unrepresented grasp types in standard taxonomies, uneven data distribution across injury levels, and limited data. This study aims to automatically identify the dominant distinct hand grasps in egocentric video using semantic clustering. Egocentric video recordings collected in the homes of 19 individual with cervical SCI were used to cluster grasping actions with semantic significance. A deep learning model integrating posture and appearance data was employed to create a personalized hand taxonomy. Quantitative analysis reveals a cluster purity of 67.6% +- 24.2% with with 18.0% +- 21.8% redundancy. Qualitative assessment revealed meaningful clusters in video content. This methodology provides a flexible and effective strategy to analyze hand function in the wild. It offers researchers and clinicians an efficient tool for evaluating hand function, aiding sensitive assessments and tailored intervention plans.