# Simultaneous Alignment and Surface Regression Using Hybrid 2D-3D Networks for 3D Coherent Layer Segmentation of Retinal OCT Images with Full and Sparse Annotations

arXiv: https://arxiv.org/abs/2312.01726 · HTML: https://mlsanity.com/paper/2312.01726/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 67 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [ReLayNet: Retinal Layer and Fluid Segmentation of Macular Optical Coherence Tomography using Fully Convolutional Network](https://mlsanity.com/paper/1704.02161/) (2017)
- [Uncertainty guided semi-supervised segmentation of retinal layers in OCT images](https://mlsanity.com/paper/2103.02083/) (2021)
- [Recurrent neural networks for aortic image sequence segmentation with sparse annotations](https://mlsanity.com/paper/1808.00273/) (2018)
- [Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss](https://mlsanity.com/paper/1906.03906/) (2019)
- [Multi-scale GCN-assisted two-stage network for joint segmentation of retinal layers and disc in peripapillary OCT images](https://mlsanity.com/paper/2102.04799/) (2021)
- [Simultaneous Alignment and Surface Regression Using Hybrid 2D-3D Networks for 3D Coherent Layer Segmentation of Retina OCT Images](https://mlsanity.com/paper/2203.02390/) (2022)
- [Conquering Data Variations in Resolution: A Slice-Aware Multi-Branch Decoder Network](https://mlsanity.com/paper/2203.03640/) (2022)

## Built on by (most cited first, 7 of 7)

- [MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis](https://mlsanity.com/paper/2506.08900/) (2025)
- [SD-RetinaNet: Topologically Constrained Semi-Supervised Retinal Lesion and Layer Segmentation in OCT](https://mlsanity.com/paper/2509.20864/) (2025)
- [Formula-Driven Data Augmentation and Partial Retinal Layer Copying for Retinal Layer Segmentation](https://mlsanity.com/paper/2410.01185/) (2024)
- [Uncertainty-aware retinal layer segmentation in OCT through probabilistic signed distance functions](https://mlsanity.com/paper/2412.04935/) (2024)
- [Learning with Geometry: Including Riemannian Geometric Features in Coefficient of Pressure Prediction on Aircraft Wings](https://mlsanity.com/paper/2401.09452/) (2023)
- [Don't Mind the Gaps: Implicit Neural Representations for Resolution-Agnostic Retinal OCT Analysis](https://mlsanity.com/paper/2601.02447/) (2026)
- [PC-Seg: Progressive Cross-View Consistency for 3D OCT Segmentation from Sparse 2D Annotations](https://mlsanity.com/paper/2607.17718/) (2026)

## Abstract (arXiv)

Layer segmentation is important to quantitative analysis of retinal optical coherence tomography (OCT). Recently, deep learning based methods have been developed to automate this task and yield remarkable performance. However, due to the large spatial gap and potential mismatch between the B-scans of an OCT volume, all of them were based on 2D segmentation of individual B-scans, which may lose the continuity and diagnostic information of the retinal layers in 3D space. Besides, most of these methods required dense annotation of the OCT volumes, which is labor-intensive and expertise-demanding. This work presents a novel framework based on hybrid 2D-3D convolutional neural networks (CNNs) to obtain continuous 3D retinal layer surfaces from OCT volumes, which works well with both full and sparse annotations. The 2D features of individual B-scans are extracted by an encoder consisting of 2D convolutions. These 2D features are then used to produce the alignment displacement vectors and layer segmentation by two 3D decoders coupled via a spatial transformer module. Two losses are proposed to utilize the retinal layers' natural property of being smooth for B-scan alignment and layer segmentation, respectively, and are the key to the semi-supervised learning with sparse annotation. The entire framework is trained end-to-end. To the best of our knowledge, this is the first work that attempts 3D retinal layer segmentation in volumetric OCT images based on CNNs. Experiments on a synthetic dataset and three public clinical datasets show that our framework can effectively align the B-scans for potential motion correction, and achieves superior performance to state-of-the-art 2D deep learning methods in terms of both layer segmentation accuracy and cross-B-scan 3D continuity in both fully and semi-supervised settings, thus offering more clinical values than previous works.