# Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem

arXiv: https://arxiv.org/abs/2301.08441 · HTML: https://mlsanity.com/paper/2301.08441/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 79 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [Self-Supervised Learning for Recommender Systems: A Survey](https://mlsanity.com/paper/2203.15876/) (2022)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [Potential, Challenges and Future Directions for Deep Learning in Prognostics and Health Management Applications](https://mlsanity.com/paper/2005.02144/) (2020)
- [S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation](https://mlsanity.com/paper/2005.11437/) (2020)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Tensorized Embedding Layers for Efficient Model Compression](https://mlsanity.com/paper/1901.10787/) (2019)
- [Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks](https://mlsanity.com/paper/1805.08569/) (2018)
- [GaitForeMer: Self-Supervised Pre-Training of Transformers via Human Motion Forecasting for Few-Shot Gait Impairment Severity Estimation](https://mlsanity.com/paper/2207.00106/) (2022)
- [Audacity of huge: overcoming challenges of data scarcity and data quality for machine learning in computational materials discovery](https://mlsanity.com/paper/2111.01905/) (2021)
- [COVID-19 detection from scarce chest x-ray image data using few-shot deep learning approach](https://mlsanity.com/paper/2102.06285/) (2021)
- [Semi-supervised Learning with Deep Generative Models for Asset Failure Prediction](https://mlsanity.com/paper/1709.00845/) (2017)
- [Masked Self-Supervision for Remaining Useful Lifetime Prediction in Machine Tools](https://mlsanity.com/paper/2207.01219/) (2022)
- [Gated Recurrent Unit (GRU) for Emotion Classification from Noisy Speech](https://mlsanity.com/paper/1612.07778/) (2016)
- [Improving Semi-Supervised Learning for Remaining Useful Lifetime Estimation Through Self-Supervision](https://mlsanity.com/paper/2108.08721/) (2021)
- [Improving Maritime Traffic Emission Estimations on Missing Data with CRBMs](https://mlsanity.com/paper/2009.03001/) (2020)

## Built on by (most cited first, 3 of 3)

- [Self-Supervised Learning for Text Recognition: A Critical Survey](https://mlsanity.com/paper/2407.19889/) (2024)
- [Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Dataset](https://mlsanity.com/paper/2506.12698/) (2025)
- [A Dual-Stream Physics-Augmented Unsupervised Architecture for Runtime Embedded Vehicle Health Monitoring](https://mlsanity.com/paper/2602.10432/) (2026)

## Abstract (arXiv)

With the increasing availability of data for Prognostics and Health Management (PHM), Deep Learning (DL) techniques are now the subject of considerable attention for this application, often achieving more accurate Remaining Useful Life (RUL) predictions. However, one of the major challenges for DL techniques resides in the difficulty of obtaining large amounts of labelled data on industrial systems. To overcome this lack of labelled data, an emerging learning technique is considered in our work: Self-Supervised Learning, a sub-category of unsupervised learning approaches. This paper aims to investigate whether pre-training DL models in a self-supervised way on unlabelled sensors data can be useful for RUL estimation with only Few-Shots Learning, i.e. with scarce labelled data. In this research, a fatigue damage prognostics problem is addressed, through the estimation of the RUL of aluminum alloy panels (typical of aerospace structures) subject to fatigue cracks from strain gauge data. Synthetic datasets composed of strain data are used allowing to extensively investigate the influence of the dataset size on the predictive performance. Results show that the self-supervised pre-trained models are able to significantly outperform the non-pre-trained models in downstream RUL prediction task, and with less computational expense, showing promising results in prognostic tasks when only limited labelled data is available.