# Image-based Freeform Handwriting Authentication with Energy-oriented Self-Supervised Learning

arXiv: https://arxiv.org/abs/2408.09676 · HTML: https://mlsanity.com/paper/2408.09676/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 93 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Learning Data Augmentation Strategies for Object Detection](https://mlsanity.com/paper/1906.11172/) (2019)
- [Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive](https://mlsanity.com/paper/2105.07342/) (2021)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [Learning Representations by Predicting Bags of Visual Words](https://mlsanity.com/paper/2002.12247/) (2020)
- [Corrupted Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2202.03382/) (2022)
- [Text and Style Conditioned GAN for Generation of Offline Handwriting Lines](https://mlsanity.com/paper/2009.00678/) (2020)
- [Handwriting Biometrics: Applications and Future Trends in e-Security and e-Health](https://mlsanity.com/paper/2202.12760/) (2022)
- [FragNet: Writer Identification using Deep Fragment Networks](https://mlsanity.com/paper/2003.07212/) (2020)
- [Flow-based Self-supervised Density Estimation for Anomalous Sound Detection](https://mlsanity.com/paper/2103.08801/) (2021)
- [GR-RNN: Global-Context Residual Recurrent Neural Networks for Writer Identification](https://mlsanity.com/paper/2104.05036/) (2021)
- [Self-Supervised Representation Learning via Neighborhood-Relational Encoding](https://mlsanity.com/paper/1908.10455/) (2019)
- [Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis](https://mlsanity.com/paper/2205.12144/) (2022)
- [SURDS: Self-Supervised Attention-guided Reconstruction and Dual Triplet Loss for Writer Independent Offline Signature Verification](https://mlsanity.com/paper/2201.10138/) (2022)
- [SWIS: Self-Supervised Representation Learning For Writer Independent Offline Signature Verification](https://mlsanity.com/paper/2202.13078/) (2022)

## Built on by (most cited first, 3 of 3)

- [DiffDesign: Controllable Diffusion with Meta Prior for Efficient Interior Design Generation](https://mlsanity.com/paper/2411.16301/) (2024)
- [Neuromodulated Meta-Learning](https://mlsanity.com/paper/2411.06746/) (2024)
- [Unsupervised Multi-Attention Meta Transformer for Rotating Machinery Fault Diagnosis](https://mlsanity.com/paper/2509.09251/) (2025)

## Abstract (arXiv)

Freeform handwriting authentication verifies a person's identity from their writing style and habits in messy handwriting data. This technique has gained widespread attention in recent years as a valuable tool for various fields, e.g., fraud prevention and cultural heritage protection. However, it still remains a challenging task in reality due to three reasons: (i) severe damage, (ii) complex high-dimensional features, and (iii) lack of supervision. To address these issues, we propose SherlockNet, an energy-oriented two-branch contrastive self-supervised learning framework for robust and fast freeform handwriting authentication. It consists of four stages: (i) pre-processing: converting manuscripts into energy distributions using a novel plug-and-play energy-oriented operator to eliminate the influence of noise; (ii) generalized pre-training: learning general representation through two-branch momentum-based adaptive contrastive learning with the energy distributions, which handles the high-dimensional features and spatial dependencies of handwriting; (iii) personalized fine-tuning: calibrating the learned knowledge using a small amount of labeled data from downstream tasks; and (iv) practical application: identifying individual handwriting from scrambled, missing, or forged data efficiently and conveniently. Considering the practicality, we construct EN-HA, a novel dataset that simulates data forgery and severe damage in real applications. Finally, we conduct extensive experiments on six benchmark datasets including our EN-HA, and the results prove the robustness and efficiency of SherlockNet.