# A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wild

arXiv: https://arxiv.org/abs/2105.07625 · HTML: https://mlsanity.com/paper/2105.07625/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Maximum-Entropy Fine-Grained Classification](https://mlsanity.com/paper/1809.05934/) (2018)
- [Fingerspelling recognition in the wild with iterative visual attention](https://mlsanity.com/paper/1908.10546/) (2019)
- [American Sign Language fingerspelling recognition in the wild](https://mlsanity.com/paper/1810.11438/) (2018)
- [SGNet: A Super-class Guided Network for Image Classification and Object Detection](https://mlsanity.com/paper/2104.12898/) (2021)
- [A Comparative Study on Polyp Classification using Convolutional Neural Networks](https://mlsanity.com/paper/2007.06071/) (2020)
- [Adaptively Denoising Proposal Collection for Weakly Supervised Object Localization](https://mlsanity.com/paper/1910.02101/) (2019)
- [Efficient Golf Ball Detection and Tracking Based on Convolutional Neural Networks and Kalman Filter](https://mlsanity.com/paper/2012.09393/) (2020)
- [Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation](https://mlsanity.com/paper/1609.07876/) (2016)
- [Boosting Occluded Image Classification via Subspace Decomposition Based Estimation of Deep Features](https://mlsanity.com/paper/2001.04066/) (2020)
- [Parallel Scale-wise Attention Network for Effective Scene Text Recognition](https://mlsanity.com/paper/2104.12076/) (2021)
- [Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition](https://mlsanity.com/paper/1710.03255/) (2017)

## Built on by (most cited first, 8 of 8)

- [Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.12333/) (2022)
- [Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergence](https://mlsanity.com/paper/2112.13310/) (2021)
- [Aggregating Global Features into Local Vision Transformer](https://mlsanity.com/paper/2201.12903/) (2022)
- [Fingerspelling PoseNet: Enhancing Fingerspelling Translation with Pose-Based Transformer Models](https://mlsanity.com/paper/2311.12128/) (2023)
- [Multiple Classifiers Based Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2108.00610/) (2021)
- [Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.14319/) (2022)
- [SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation](https://mlsanity.com/paper/2509.00030/) (2025)
- [OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis](https://mlsanity.com/paper/2602.22949/) (2026)

## Abstract (arXiv)

Fingerspelling in sign language has been the means of communicating technical terms and proper nouns when they do not have dedicated sign language gestures. Automatic recognition of fingerspelling can help resolve communication barriers when interacting with deaf people. The main challenges prevalent in fingerspelling recognition are the ambiguity in the gestures and strong articulation of the hands. The automatic recognition model should address high inter-class visual similarity and high intra-class variation in the gestures. Most of the existing research in fingerspelling recognition has focused on the dataset collected in a controlled environment. The recent collection of a large-scale annotated fingerspelling dataset in the wild, from social media and online platforms, captures the challenges in a real-world scenario. In this work, we propose a fine-grained visual attention mechanism using the Transformer model for the sequence-to-sequence prediction task in the wild dataset. The fine-grained attention is achieved by utilizing the change in motion of the video frames (optical flow) in sequential context-based attention along with a Transformer encoder model. The unsegmented continuous video dataset is jointly trained by balancing the Connectionist Temporal Classification (CTC) loss and the maximum-entropy loss. The proposed approach can capture better fine-grained attention in a single iteration. Experiment evaluations show that it outperforms the state-of-the-art approaches.