# Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

arXiv: https://arxiv.org/abs/2301.07743 · HTML: https://mlsanity.com/paper/2301.07743/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 53 references that are in the map.
- First posted on arXiv 18 Jan 2023.

## Builds on (most cited first, 12 of 12)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Review: Deep Learning in Electron Microscopy](https://mlsanity.com/paper/2009.08328/) (2020)
- [Improving Shape Deformation in Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1808.04325/) (2018)
- [Paradigm shift in electron-based crystallography via machine learning](https://mlsanity.com/paper/1902.03682/) (2019)
- [An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics](https://mlsanity.com/paper/2109.14772/) (2021)
- [Disentangling multiple scattering with deep learning: application to strain mapping from electron diffraction patterns](https://mlsanity.com/paper/2202.00204/) (2022)
- [Image registration of low signal-to-noise cryo-STEM data](https://mlsanity.com/paper/1710.09281/) (2017)
- [Deep Snow: Synthesizing Remote Sensing Imagery with Generative Adversarial Nets](https://mlsanity.com/paper/2005.08892/) (2020)
- [Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows](https://mlsanity.com/paper/2208.02865/) (2022)

## Built on by (most cited first, 8 of 8)

- [Noise Calibration and Spatial-Frequency Interactive Network for STEM Image Enhancement](https://mlsanity.com/paper/2504.02555/) (2025)
- [SAM-I-Am: Semantic Boosting for Zero-shot Atomic-Scale Electron Micrograph Segmentation](https://mlsanity.com/paper/2404.06638/) (2024)
- [A robust synthetic data generation framework for machine learning in High-Resolution Transmission Electron Microscopy (HRTEM)](https://mlsanity.com/paper/2309.06122/) (2023)
- [Statistical Characteristic-Guided Denoising for Rapid High-Resolution Transmission Electron Microscopy Imaging](https://mlsanity.com/paper/2603.18834/) (2026)
- [Overcoming Labelled Data Scarcity for Defect Classification in Scanning Tunneling Microscopy](https://mlsanity.com/paper/2506.01678/) (2025)
- [Contrast transfer functions help quantify neural network out-of-distribution generalization in HRTEM](https://mlsanity.com/paper/2512.09067/) (2025)
- [Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy](https://mlsanity.com/paper/2604.24909/) (2026)
- [High-Fidelity Synthetic Transmission Electron Microscopy Image Generation Using Diffusion Probabilistic Models for Data-Limited Semiconductor Metrology](https://mlsanity.com/paper/2606.24817/) (2026)

## Abstract (arXiv)

The rise of automation and machine learning (ML) in electron microscopy has the potential to revolutionize materials research through autonomous data collection and processing. A significant challenge lies in developing ML models that rapidly generalize to large data sets under varying experimental conditions. We address this by employing a cycle generative adversarial network (CycleGAN) with a reciprocal space discriminator, which augments simulated data with realistic spatial frequency information. This allows the CycleGAN to generate images nearly indistinguishable from real data and provide labels for ML applications. We showcase our approach by training a fully convolutional network (FCN) to identify single atom defects in a 4.5 million atom data set, collected using automated acquisition in an aberration-corrected scanning transmission electron microscope (STEM). Our method produces adaptable FCNs that can adjust to dynamically changing experimental variables with minimal intervention, marking a crucial step towards fully autonomous harnessing of microscopy big data.