# The Quest for Early Detection of Retinal Disease: 3D CycleGAN-based Translation of Optical Coherence Tomography into Confocal Microscopy

arXiv: https://arxiv.org/abs/2408.04091 · HTML: https://mlsanity.com/paper/2408.04091/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 76 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Unsupervised Cross-Domain Image Generation](https://mlsanity.com/paper/1611.02200/) (2016)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Unpaired Image-to-Image Translation via Neural Schr"odinger Bridge](https://mlsanity.com/paper/2305.15086/) (2023)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [Efficient Active Learning for Image Classification and Segmentation using a Sample Selection and Conditional Generative Adversarial Network](https://mlsanity.com/paper/1806.05473/) (2018)
- [Towards Deeper Understanding of Variational Autoencoding Models](https://mlsanity.com/paper/1702.08658/) (2017)
- [Generative Adversarial Networks for Image-to-Image Translation on Multi-Contrast MR Images - A Comparison of CycleGAN and UNIT](https://mlsanity.com/paper/1806.07777/) (2018)
- [Multimodal Transfer Learning-based Approaches for Retinal Vascular Segmentation](https://mlsanity.com/paper/2012.10160/) (2020)
- [Image-to-Image Translation for Autonomous Driving from Coarsely-Aligned Image Pairs](https://mlsanity.com/paper/2209.11673/) (2022)
- [Unpaired Image-to-Image Translation with Limited Data to Reveal Subtle Phenotypes](https://mlsanity.com/paper/2302.08503/) (2023)
- [Adversarial Sparse-View CBCT Artifact Reduction](https://mlsanity.com/paper/1812.03503/) (2018)
- [Optimal Transport-based Graph Matching for 3D retinal OCT image registration](https://mlsanity.com/paper/2203.00069/) (2022)
- [Objective quality assessment of medical images and videos: Review and challenges](https://mlsanity.com/paper/2212.07396/) (2022)
- [OCT2Confocal: 3D CycleGAN based Translation of Retinal OCT Images to Confocal Microscopy](https://mlsanity.com/paper/2311.10902/) (2023)
- [Generating Fundus Fluorescence Angiography Images from Structure Fundus Images Using Generative Adversarial Networks](https://mlsanity.com/paper/2006.10216/) (2020)

## Abstract (arXiv)

Optical coherence tomography (OCT) and confocal microscopy are pivotal in retinal imaging, offering distinct advantages and limitations. In vivo OCT offers rapid, non-invasive imaging but can suffer from clarity issues and motion artifacts, while ex vivo confocal microscopy, providing high-resolution, cellular-detailed color images, is invasive and raises ethical concerns. To bridge the benefits of both modalities, we propose a novel framework based on unsupervised 3D CycleGAN for translating unpaired in vivo OCT to ex vivo confocal microscopy images. This marks the first attempt to exploit the inherent 3D information of OCT and translate it into the rich, detailed color domain of confocal microscopy. We also introduce a unique dataset, OCT2Confocal, comprising mouse OCT and confocal retinal images, facilitating the development of and establishing a benchmark for cross-modal image translation research. Our model has been evaluated both quantitatively and qualitatively, achieving Fr'echet Inception Distance (FID) scores of 0.766 and Kernel Inception Distance (KID) scores as low as 0.153, and leading subjective Mean Opinion Scores (MOS). Our model demonstrated superior image fidelity and quality with limited data over existing methods. Our approach effectively synthesizes color information from 3D confocal images, closely approximating target outcomes and suggesting enhanced potential for diagnostic and monitoring applications in ophthalmology.