# Deep Learning of Crystalline Defects from TEM images: A Solution for the Problem of "Never Enough Training Data"

arXiv: https://arxiv.org/abs/2307.06322 · HTML: https://mlsanity.com/paper/2307.06322/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 57 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images](https://mlsanity.com/paper/1805.06561/) (2018)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [Deep Learning in Medical Image Registration: A Review](https://mlsanity.com/paper/1912.12318/) (2019)
- [SinGAN-Seg: Synthetic training data generation for medical image segmentation](https://mlsanity.com/paper/2107.00471/) (2021)
- [Learning Depth from Single Images with Deep Neural Network Embedding Focal Length](https://mlsanity.com/paper/1803.10039/) (2018)
- [Physics-driven Synthetic Data Learning for Biomedical Magnetic Resonance](https://mlsanity.com/paper/2203.11178/) (2022)
- [Deep Leaf Segmentation Using Synthetic Data](https://mlsanity.com/paper/1807.10931/) (2018)
- [Deep learning for synthetic microstructure generation in a materials-by-design framework for heterogeneous energetic materials](https://mlsanity.com/paper/2004.04814/) (2020)
- [Synthetic Image Rendering Solves Annotation Problem in Deep Learning Nanoparticle Segmentation](https://mlsanity.com/paper/2011.10505/) (2020)
- [Multi defect detection and analysis of electron microscopy images with deep learning](https://mlsanity.com/paper/2108.08883/) (2021)
- [Initialization Using Perlin Noise for Training Networks with a Limited Amount of Data](https://mlsanity.com/paper/2101.07406/) (2021)

## Built on by (most cited first, 4 of 4)

- [Self-Supervised Learning with Generative Adversarial Networks for Electron Microscopy](https://mlsanity.com/paper/2402.18286/) (2024)
- [Modeling Dislocation Dynamics Data Using Semantic Web Technologies](https://mlsanity.com/paper/2309.06930/) (2023)
- [Combining unsupervised and supervised learning in microscopy enables defect analysis of a full 4H-SiC wafer](https://mlsanity.com/paper/2402.13353/) (2024)
- [High-Fidelity Synthetic Transmission Electron Microscopy Image Generation Using Diffusion Probabilistic Models for Data-Limited Semiconductor Metrology](https://mlsanity.com/paper/2606.24817/) (2026)

## Abstract (arXiv)

Crystalline defects, such as line-like dislocations, play an important role for the performance and reliability of many metallic devices. Their interaction and evolution still poses a multitude of open questions to materials science and materials physics. In-situ TEM experiments can provide important insights into how dislocations behave and move. During such experiments, the dislocation microstructure is captured in form of videos. The analysis of individual video frames can provide useful insights but is limited by the capabilities of automated identification, digitization, and quantitative extraction of the dislocations as curved objects. The vast amount of data also makes manual annotation very time consuming, thereby limiting the use of Deep Learning-based, automated image analysis and segmentation of the dislocation microstructure. In this work, a parametric model for generating synthetic training data for segmentation of dislocations is developed. Even though domain scientists might dismiss synthetic training images sometimes as too artificial, our findings show that they can result in superior performance, particularly regarding the generalizing of the Deep Learning models with respect to different microstructures and imaging conditions. Additionally, we propose an enhanced deep learning method optimized for segmenting overlapping or intersecting dislocation lines. Upon testing this framework on four distinct real datasets, we find that our synthetic training data are able to yield high-quality results also on real images-even more so if fine-tune on a few real images was done.