# A Transformer Architecture for Online Gesture Recognition of Mathematical Expressions

arXiv: https://arxiv.org/abs/2211.02643 · HTML: https://mlsanity.com/paper/2211.02643/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention](https://mlsanity.com/paper/2007.10092/) (2020)
- [Character-Based Handwritten Text Transcription with Attention Networks](https://mlsanity.com/paper/1712.04046/) (2017)
- [Text Line Segmentation for Challenging Handwritten Document Images Using Fully Convolutional Network](https://mlsanity.com/paper/2101.08299/) (2021)
- [Lightweight Network Architecture for Real-Time Action Recognition](https://mlsanity.com/paper/1905.08711/) (2019)
- [Open Source Dataset and Deep Learning Models for Online Digit Gesture Recognition on Touchscreens](https://mlsanity.com/paper/1709.06871/) (2017)

## Abstract (arXiv)

The Transformer architecture is shown to provide a powerful framework as an end-to-end model for building expression trees from online handwritten gestures corresponding to glyph strokes. In particular, the attention mechanism was successfully used to encode, learn and enforce the underlying syntax of expressions creating latent representations that are correctly decoded to the exact mathematical expression tree, providing robustness to ablated inputs and unseen glyphs. For the first time, the encoder is fed with spatio-temporal data tokens potentially forming an infinitely large vocabulary, which finds applications beyond that of online gesture recognition. A new supervised dataset of online handwriting gestures is provided for training models on generic handwriting recognition tasks and a new metric is proposed for the evaluation of the syntactic correctness of the output expression trees. A small Transformer model suitable for edge inference was successfully trained to an average normalised Levenshtein accuracy of 94%, resulting in valid postfix RPN tree representation for 94% of predictions.