# Leveraging Expert Models for Training Deep Neural Networks in Scarce Data Domains: Application to Offline Handwritten Signature Verification

arXiv: https://arxiv.org/abs/2308.01136 · HTML: https://mlsanity.com/paper/2308.01136/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 98 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [SigNet: Convolutional Siamese Network for Writer Independent Offline Signature Verification](https://mlsanity.com/paper/1707.02131/) (2017)
- [Learning Features for Offline Handwritten Signature Verification using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1705.05787/) (2017)
- [Offline Handwritten Signature Verification - Literature Review](https://mlsanity.com/paper/1507.07909/) (2015)
- [Characterizing and evaluating adversarial examples for Offline Handwritten Signature Verification](https://mlsanity.com/paper/1901.03398/) (2019)
- [SURDS: Self-Supervised Attention-guided Reconstruction and Dual Triplet Loss for Writer Independent Offline Signature Verification](https://mlsanity.com/paper/2201.10138/) (2022)
- [SWIS: Self-Supervised Representation Learning For Writer Independent Offline Signature Verification](https://mlsanity.com/paper/2202.13078/) (2022)
- [Fixed-sized representation learning from Offline Handwritten Signatures of different sizes](https://mlsanity.com/paper/1804.00448/) (2018)
- [Offline Signature Verification on Real-World Documents](https://mlsanity.com/paper/2004.12104/) (2020)
- [Signature Verification Approach using Fusion of Hybrid Texture Features](https://mlsanity.com/paper/1709.09348/) (2017)
- [A comprehensive study of sparse representation techniques for offline signature verification](https://mlsanity.com/paper/1807.05039/) (2018)
- [Intrapersonal Parameter Optimization for Offline Handwritten Signature Augmentation](https://mlsanity.com/paper/2010.06663/) (2020)
- [Learning Representations from Persian Handwriting for Offline Signature Verification, a Deep Transfer Learning Approach](https://mlsanity.com/paper/1903.06249/) (2019)
- [Multi-Representational Learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs](https://mlsanity.com/paper/1903.06536/) (2019)

## Abstract (arXiv)

This paper introduces a novel approach to leverage the knowledge of existing expert models for training new Convolutional Neural Networks, on domains where task-specific data are limited or unavailable. The presented scheme is applied in offline handwritten signature verification (OffSV) which, akin to other biometric applications, suffers from inherent data limitations due to regulatory restrictions. The proposed Student-Teacher (S-T) configuration utilizes feature-based knowledge distillation (FKD), combining graph-based similarity for local activations with global similarity measures to supervise student's training, using only handwritten text data. Remarkably, the models trained using this technique exhibit comparable, if not superior, performance to the teacher model across three popular signature datasets. More importantly, these results are attained without employing any signatures during the feature extraction training process. This study demonstrates the efficacy of leveraging existing expert models to overcome data scarcity challenges in OffSV and potentially other related domains.