# Deep learning at the edge enables real-time streaming ptychographic imaging

arXiv: https://arxiv.org/abs/2209.09408 · HTML: https://mlsanity.com/paper/2209.09408/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [funcX: A Federated Function Serving Fabric for Science](https://mlsanity.com/paper/2005.04215/) (2020)
- [Low Photon Count Phase Retrieval Using Deep Learning](https://mlsanity.com/paper/1806.10029/) (2018)
- [AutoPhaseNN: Unsupervised Physics-aware Deep Learning of 3D Nanoscale Bragg Coherent Diffraction Imaging](https://mlsanity.com/paper/2109.14053/) (2021)
- [BraggNN: Fast X-ray Bragg Peak Analysis Using Deep Learning](https://mlsanity.com/paper/2008.08198/) (2020)
- [Three-dimensional Coherent X-ray Diffraction Imaging via Deep Convolutional Neural Networks](https://mlsanity.com/paper/2103.00001/) (2021)
- [SHARP: a distributed, GPU-based ptychographic solver](https://mlsanity.com/paper/1602.01448/) (2016)
- [Scalable and accurate multi-GPU based image reconstruction of large-scale ptychography data](https://mlsanity.com/paper/2106.07575/) (2021)
- [Real-time 3D Nanoscale Coherent Imaging via Physics-aware Deep Learning](https://mlsanity.com/paper/2006.09441/) (2020)
- [PyNX: high performance computing toolkit for coherent X-ray imaging based on operators](https://mlsanity.com/paper/2008.11511/) (2020)
- [fairDMS: Rapid Model Training by Data and Model Reuse](https://mlsanity.com/paper/2204.09805/) (2022)

## Built on by (most cited first, 17 of 17)

- [PtychoFormer: A Transformer-based Model for Ptychographic Phase Retrieval](https://mlsanity.com/paper/2410.17377/) (2024)
- [A Physics-Inspired Deep Learning Framework with Polar Coordinate Attention for Ptychographic Imaging](https://mlsanity.com/paper/2412.06806/) (2024)
- [Predicting ptychography probe positions using single-shot phase retrieval neural network](https://mlsanity.com/paper/2405.20910/) (2024)
- [NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression](https://mlsanity.com/paper/2409.05785/) (2024)
- [Ptychographic Image Reconstruction from Limited Data via Score-Based Diffusion Models with Physics-Guidance](https://mlsanity.com/paper/2502.18767/) (2025)
- [Neural Network Methods for Radiation Detectors and Imaging](https://mlsanity.com/paper/2311.05726/) (2023)
- [AI-assisted Automated Workflow for Real-time X-ray Ptychography Data Analysis via Federated Resources](https://mlsanity.com/paper/2304.04297/) (2023)
- [DONUT: Physics-aware Machine Learning for Real-time X-ray Nanodiffraction Analysis](https://mlsanity.com/paper/2507.14038/) (2025)
- [Learning neural representations for X-ray ptychography reconstruction with unknown probes](https://mlsanity.com/paper/2509.04402/) (2025)
- [Towards generalizable deep ptychography neural networks](https://mlsanity.com/paper/2509.25104/) (2025)
- [Circular Phase Representation and Geometry-Aware Optimization for Ptychographic Image Reconstruction](https://mlsanity.com/paper/2604.26664/) (2026)
- [From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures](https://mlsanity.com/paper/2509.24030/) (2025)
- [A unified self-supervised framework for single-frame Fresnel CDI and overlapped ptychography](https://mlsanity.com/paper/2602.21361/) (2026)
- [Machine Learning-Augmented Acceleration of Iterative Ptychographic Reconstruction](https://mlsanity.com/paper/2605.01122/) (2026)
- [Toward Trustworthy Autonomous Science: A Two-Year Community Roadmap](https://mlsanity.com/paper/2607.12113/) (2026)
- [Zero-Shot Cross-Material Ptychographic Phase Reconstruction Using Deep Learning](https://mlsanity.com/paper/2609.13969/) (2026)
- [Real-Time Atomic-Resolution Electron Phase Imaging without Probe Calibration via Ptychography-Supervised Learning](https://mlsanity.com/paper/2609.25684/) (2026)

## Abstract (arXiv)

Coherent microscopy techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent X-ray microscopy methods like ptychography are poised to revolutionize nanoscale materials characterization. However, associated significant increases in data and compute needs mean that conventional approaches no longer suffice for recovering sample images in real-time from high-speed coherent imaging experiments. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the sampling constraints imposed by traditional ptychography, allowing low dose imaging using orders of magnitude less data than required by traditional methods.