# EmMixformer: Mix transformer for eye movement recognition

arXiv: https://arxiv.org/abs/2401.04956 · HTML: https://mlsanity.com/paper/2401.04956/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 70 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [Fingerprint Spoof Buster](https://mlsanity.com/paper/1712.04489/) (2017)
- [Deep Fourier Up-Sampling](https://mlsanity.com/paper/2210.05171/) (2022)
- [Deep Eyedentification: Biometric Identification using Micro-Movements of the Eye](https://mlsanity.com/paper/1906.11889/) (2019)
- [GazeBase: A Large-Scale, Multi-Stimulus, Longitudinal Eye Movement Dataset](https://mlsanity.com/paper/2009.06171/) (2020)
- [Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator](https://mlsanity.com/paper/2305.15099/) (2023)
- [Delving Deep into One-Shot Skeleton-based Action Recognition with Diverse Occlusions](https://mlsanity.com/paper/2202.11423/) (2022)
- [Fourier Image Transformer](https://mlsanity.com/paper/2104.02555/) (2021)
- [Eye Know You: Metric Learning for End-to-end Biometric Authentication Using Eye Movements from a Longitudinal Dataset](https://mlsanity.com/paper/2104.10489/) (2021)
- [A Score-level Fusion Method for Eye Movement Biometrics](https://mlsanity.com/paper/1601.03333/) (2016)
- [Local-Global Transformer Enhanced Unfolding Network for Pan-sharpening](https://mlsanity.com/paper/2304.14612/) (2023)
- [TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding](https://mlsanity.com/paper/2003.07000/) (2020)
- [Eye Know You Too: A DenseNet Architecture for End-to-end Eye Movement Biometrics](https://mlsanity.com/paper/2201.02110/) (2022)
- [Short-Term Memory Convolutions](https://mlsanity.com/paper/2302.04331/) (2023)

## Built on by (most cited first, 9 of 9)

- [StarLKNet: Star Mixup with Large Kernel Networks for Palm Vein Identification](https://mlsanity.com/paper/2405.12721/) (2024)
- [Relax DARTS: Relaxing the Constraints of Differentiable Architecture Search for Eye Movement Recognition](https://mlsanity.com/paper/2409.11652/) (2024)
- [Neural Architecture Search based Global-local Vision Mamba for Palm-Vein Recognition](https://mlsanity.com/paper/2408.05743/) (2024)
- [Ocular Authentication: Fusion of Gaze and Periocular Modalities](https://mlsanity.com/paper/2505.17343/) (2025)
- [EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition](https://mlsanity.com/paper/2409.14432/) (2024)
- [Gaze Authentication: Factors Influencing Authentication Performance](https://mlsanity.com/paper/2509.10969/) (2025)
- [Detecting Autism Spectrum Disorder with Deep Eye Movement Features](https://mlsanity.com/paper/2601.05812/) (2026)
- [Modality vs. Morphology: A Framework for Time Series Classification for Biological Signals](https://mlsanity.com/paper/2605.18483/) (2026)
- [AOI-Net: Structural Face AOI-Guided Eye-Gaze Track Representation Learning for Autism Spectrum Disorder Detection](https://mlsanity.com/paper/2608.29289/) (2026)

## Abstract (arXiv)

Eye movement (EM) is a new highly secure biometric behavioral modality that has received increasing attention in recent years. Although deep neural networks, such as convolutional neural network (CNN), have recently achieved promising performance, current solutions fail to capture local and global temporal dependencies within eye movement data. To overcome this problem, we propose in this paper a mixed transformer termed EmMixformer to extract time and frequency domain information for eye movement recognition. To this end, we propose a mixed block consisting of three modules, transformer, attention Long short-term memory (attention LSTM), and Fourier transformer. We are the first to attempt leveraging transformer to learn long temporal dependencies within eye movement. Second, we incorporate the attention mechanism into LSTM to propose attention LSTM with the aim to learn short temporal dependencies. Third, we perform self attention in the frequency domain to learn global features. As the three modules provide complementary feature representations in terms of local and global dependencies, the proposed EmMixformer is capable of improving recognition accuracy. The experimental results on our eye movement dataset and two public eye movement datasets show that the proposed EmMixformer outperforms the state of the art by achieving the lowest verification error.