# On the Pitfalls of Learning with Limited Data: A Facial Expression Recognition Case Study

arXiv: https://arxiv.org/abs/2104.02653 · HTML: https://mlsanity.com/paper/2104.02653/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?](https://mlsanity.com/paper/1706.00712/) (2017)
- [Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning](https://mlsanity.com/paper/1602.03409/) (2016)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Implicit Semantic Data Augmentation for Deep Networks](https://mlsanity.com/paper/1909.12220/) (2019)
- [Learning to Compose Domain-Specific Transformations for Data Augmentation](https://mlsanity.com/paper/1709.01643/) (2017)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [ReenactGAN: Learning to Reenact Faces via Boundary Transfer](https://mlsanity.com/paper/1807.11079/) (2018)
- [Learning Character-Agnostic Motion for Motion Retargeting in 2D](https://mlsanity.com/paper/1905.01680/) (2019)
- [Going Deeper in Facial Expression Recognition using Deep Neural Networks](https://mlsanity.com/paper/1511.04110/) (2015)
- [Spatial-Temporal Recurrent Neural Network for Emotion Recognition](https://mlsanity.com/paper/1705.04515/) (2017)
- [Covariance Pooling For Facial Expression Recognition](https://mlsanity.com/paper/1805.04855/) (2018)
- [Learning to Forecast and Refine Residual Motion for Image-to-Video Generation](https://mlsanity.com/paper/1807.09951/) (2018)
- [FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition](https://mlsanity.com/paper/1609.06591/) (2016)
- [A Survey on Face Data Augmentation](https://mlsanity.com/paper/1904.11685/) (2019)
- [Peak-Piloted Deep Network for Facial Expression Recognition](https://mlsanity.com/paper/1607.06997/) (2016)
- [DADA: Deep Adversarial Data Augmentation for Extremely Low Data Regime Classification](https://mlsanity.com/paper/1809.00981/) (2018)
- [Infinite Brain MR Images: PGGAN-based Data Augmentation for Tumor Detection](https://mlsanity.com/paper/1903.12564/) (2019)
- [Learning More with Less: Conditional PGGAN-based Data Augmentation for Brain Metastases Detection Using Highly-Rough Annotation on MR Images](https://mlsanity.com/paper/1902.09856/) (2019)
- [Spatio-Temporal Facial Expression Recognition Using Convolutional Neural Networks and Conditional Random Fields](https://mlsanity.com/paper/1703.06995/) (2017)
- [Recognition of facial expressions based on salient geometric features and support vector machines](https://mlsanity.com/paper/1604.04334/) (2016)

## Abstract (arXiv)

Deep learning models need large amounts of data for training. In video recognition and classification, significant advances were achieved with the introduction of new large databases. However, the creation of large-databases for training is infeasible in several scenarios. Thus, existing or small collected databases are typically joined and amplified to train these models. Nevertheless, training neural networks on limited data is not straightforward and comes with a set of problems. In this paper, we explore the effects of stacking databases, model initialization, and data amplification techniques when training with limited data on deep learning models' performance. We focused on the problem of Facial Expression Recognition from videos. We performed an extensive study with four databases at a different complexity and nine deep-learning architectures for video classification. We found that (i) complex training sets translate better to more stable test sets when trained with transfer learning and synthetically generated data, but their performance yields a high variance; (ii) training with more detailed data translates to more stable performance on novel scenarios (albeit with lower performance); (iii) merging heterogeneous data is not a straightforward improvement, as the type of augmentation and initialization is crucial; (iv) classical data augmentation cannot fill the holes created by joining largely separated datasets; and (v) inductive biases help to bridge the gap when paired with synthetic data, but this data is not enough when working with standard initialization techniques.