# Writing in The Air: Unconstrained Text Recognition from Finger Movement Using Spatio-Temporal Convolution

arXiv: https://arxiv.org/abs/2104.09021 · HTML: https://mlsanity.com/paper/2104.09021/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 83 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)
- [OrigamiNet: Weakly-Supervised, Segmentation-Free, One-Step, Full Page Text Recognition by learning to unfold](https://mlsanity.com/paper/2006.07491/) (2020)
- [MVFNet: Multi-View Fusion Network for Efficient Video Recognition](https://mlsanity.com/paper/2012.06977/) (2020)
- [Fast Spatio-Temporal Residual Network for Video Super-Resolution](https://mlsanity.com/paper/1904.02870/) (2019)
- [Drawing and Recognizing Chinese Characters with Recurrent Neural Network](https://mlsanity.com/paper/1606.06539/) (2016)
- [Adaptive pooling operators for weakly labeled sound event detection](https://mlsanity.com/paper/1804.10070/) (2018)
- [Fast Multi-language LSTM-based Online Handwriting Recognition](https://mlsanity.com/paper/1902.10525/) (2019)
- [Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition](https://mlsanity.com/paper/1610.02616/) (2016)
- [Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition](https://mlsanity.com/paper/1704.07160/) (2017)
- [Fingertip Detection and Tracking for Recognition of Air-Writing in Videos](https://mlsanity.com/paper/1809.03016/) (2018)
- [Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks](https://mlsanity.com/paper/2101.08919/) (2021)
- [Convolutional Recurrent Reconstructive Network for Spatiotemporal Anomaly Detection in Solder Paste Inspection](https://mlsanity.com/paper/1908.08204/) (2019)
- [AirDraw: Leveraging Smart Watch Motion Sensors for Mobile Human Computer Interactions](https://mlsanity.com/paper/1705.02689/) (2017)
- [I-Keyboard: Fully Imaginary Keyboard on Touch Devices Empowered by Deep Neural Decoder](https://mlsanity.com/paper/1907.13285/) (2019)

## Built on by (most cited first, 1 of 1)

- [Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World](https://mlsanity.com/paper/2412.19537/) (2024)

## Abstract (arXiv)

In this paper, we introduce a new benchmark dataset for the challenging writing in the air (WiTA) task -- an elaborate task bridging vision and NLP. WiTA implements an intuitive and natural writing method with finger movement for human-computer interaction (HCI). Our WiTA dataset will facilitate the development of data-driven WiTA systems which thus far have displayed unsatisfactory performance -- due to lack of dataset as well as traditional statistical models they have adopted. Our dataset consists of five sub-datasets in two languages (Korean and English) and amounts to 209,926 video instances from 122 participants. We capture finger movement for WiTA with RGB cameras to ensure wide accessibility and cost-efficiency. Next, we propose spatio-temporal residual network architectures inspired by 3D ResNet. These models perform unconstrained text recognition from finger movement, guarantee a real-time operation by processing 435 and 697 decoding frames-per-second for Korean and English, respectively, and will serve as an evaluation standard. Our dataset and the source codes are available at https://github.com/Uehwan/WiTA.