# Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation

arXiv: https://arxiv.org/abs/1704.07754 · HTML: https://mlsanity.com/paper/1704.07754/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2017.

## Built on by (most cited first, 15 of 15)

- [mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation](https://mlsanity.com/paper/2206.02425/) (2022)
- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation](https://mlsanity.com/paper/2207.07370/) (2022)
- [Multi-Modality Cardiac Image Computing: A Survey](https://mlsanity.com/paper/2208.12881/) (2022)
- [A Tri-attention Fusion Guided Multi-modal Segmentation Network](https://mlsanity.com/paper/2111.01623/) (2021)
- [Glioblastoma Multiforme Prognosis: MRI Missing Modality Generation, Segmentation and Radiogenomic Survival Prediction](https://mlsanity.com/paper/2104.01149/) (2021)
- [DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2505.11921/) (2025)
- [3D Medical Multi-modal Segmentation Network Guided by Multi-source Correlation Constraint](https://mlsanity.com/paper/2102.03111/) (2021)
- [Multiple Sclerosis Lesions Segmentation using Attention-Based CNNs in FLAIR Images](https://mlsanity.com/paper/2201.01832/) (2022)
- [U-Net-and-a-half: Convolutional network for biomedical image segmentation using multiple expert-driven annotations](https://mlsanity.com/paper/2108.04658/) (2021)
- [Recurrent U-net for automatic pelvic floor muscle segmentation on 3D ultrasound](https://mlsanity.com/paper/2107.13833/) (2021)
- [Mutual Attention-based Hybrid Dimensional Network for Multimodal Imaging Computer-aided Diagnosis](https://mlsanity.com/paper/2201.09421/) (2022)
- [Two-stage MR Image Segmentation Method for Brain Tumors based on Attention Mechanism](https://mlsanity.com/paper/2304.08072/) (2023)
- [Cloud-based Federated Learning Framework for MRI Segmentation](https://mlsanity.com/paper/2403.00254/) (2024)
- [Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2604.22177/) (2026)

## Abstract (arXiv)

Deep learning models such as convolutional neural net- work have been widely used in 3D biomedical segmentation and achieve state-of-the-art performance. However, most of them often adapt a single modality or stack multiple modalities as different input channels. To better leverage the multi- modalities, we propose a deep encoder-decoder structure with cross-modality convolution layers to incorporate different modalities of MRI data. In addition, we exploit convolutional LSTM to model a sequence of 2D slices, and jointly learn the multi-modalities and convolutional LSTM in an end-to-end manner. To avoid converging to the certain labels, we adopt a re-weighting scheme and two-phase training to handle the label imbalance. Experimental results on BRATS-2015 show that our method outperforms state-of-the-art biomedical segmentation approaches.