# Combiner and HyperCombiner Networks: Rules to Combine Multimodality MR Images for Prostate Cancer Localisation

arXiv: https://arxiv.org/abs/2307.08279 · HTML: https://mlsanity.com/paper/2307.08279/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 74 references that are in the map.
- First posted on arXiv 17 Jul 2023.

## Builds on (most cited first, 14 of 14)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [HyperMorph: Amortized Hyperparameter Learning for Image Registration](https://mlsanity.com/paper/2101.01035/) (2021)
- [Stochastic Hyperparameter Optimization through Hypernetworks](https://mlsanity.com/paper/1802.09419/) (2018)
- [Hypernetwork functional image representation](https://mlsanity.com/paper/1902.10404/) (2019)
- [End-to-end Prostate Cancer Detection in bpMRI via 3D CNNs: Effects of Attention Mechanisms, Clinical Priori and Decoupled False Positive Reduction](https://mlsanity.com/paper/2101.03244/) (2021)
- [ProstAttention-Net: A deep attention model for prostate cancer segmentation by aggressiveness in MRI scans](https://mlsanity.com/paper/2211.13238/) (2022)
- [Hyper-Convolution Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/2105.10559/) (2021)
- [Harnessing Uncertainty in Domain Adaptation for MRI Prostate Lesion Segmentation](https://mlsanity.com/paper/2010.07411/) (2020)
- [Regularization-Agnostic Compressed Sensing MRI Reconstruction with Hypernetworks](https://mlsanity.com/paper/2101.02194/) (2021)
- [Image quality assessment by overlapping task-specific and task-agnostic measures: application to prostate multiparametric MR images for cancer segmentation](https://mlsanity.com/paper/2202.09798/) (2022)
- [The impact of using voxel-level segmentation metrics on evaluating multifocal prostate cancer localisation](https://mlsanity.com/paper/2203.16415/) (2022)

## Built on by (most cited first, 10 of 10)

- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Poisson Ordinal Network for Gleason Group Estimation Using Bi-Parametric MRI](https://mlsanity.com/paper/2407.05796/) (2024)
- [T2-Only Prostate Cancer Prediction by Meta-Learning from Bi-Parametric MR Imaging](https://mlsanity.com/paper/2411.07416/) (2024)
- [ProFound: A moderate-sized vision foundation model for multi-task prostate imaging](https://mlsanity.com/paper/2603.03961/) (2026)
- [Promptable segmentation with region exploration enables minimal-effort expert-level prostate cancer delineation](https://mlsanity.com/paper/2602.17813/) (2026)
- [Deep EM with Hierarchical Latent Label Modelling for Multi-Site Prostate Lesion Segmentation](https://mlsanity.com/paper/2603.14418/) (2026)
- [On the Degrees of Freedom of Gridded Control Points in Learning-Based Medical Image Registration](https://mlsanity.com/paper/2603.16940/) (2026)
- [Maximizing T2-Only Prostate Cancer Localization from Expected Diffusion Weighted Imaging](https://mlsanity.com/paper/2604.00985/) (2026)
- [Learning to Distort: Weakly-Supervised Image Quality Transfer for Prostate DWI Correction](https://mlsanity.com/paper/2606.18869/) (2026)
- [Bridging Single Distortion Artifacts and Multifactorial Clinical Quality: Few-shot Biparametric MRI Quality Assessment via Distortion-trained Prototypical Networks](https://mlsanity.com/paper/2606.18872/) (2026)

## Abstract (arXiv)

One of the distinct characteristics in radiologists' reading of multiparametric prostate MR scans, using reporting systems such as PI-RADS v2.1, is to score individual types of MR modalities, T2-weighted, diffusion-weighted, and dynamic contrast-enhanced, and then combine these image-modality-specific scores using standardised decision rules to predict the likelihood of clinically significant cancer. This work aims to demonstrate that it is feasible for low-dimensional parametric models to model such decision rules in the proposed Combiner networks, without compromising the accuracy of predicting radiologic labels: First, it is shown that either a linear mixture model or a nonlinear stacking model is sufficient to model PI-RADS decision rules for localising prostate cancer. Second, parameters of these (generalised) linear models are proposed as hyperparameters, to weigh multiple networks that independently represent individual image modalities in the Combiner network training, as opposed to end-to-end modality ensemble. A HyperCombiner network is developed to train a single image segmentation network that can be conditioned on these hyperparameters during inference, for much improved efficiency. Experimental results based on data from 850 patients, for the application of automating radiologist labelling multi-parametric MR, compare the proposed combiner networks with other commonly-adopted end-to-end networks. Using the added advantages of obtaining and interpreting the modality combining rules, in terms of the linear weights or odds-ratios on individual image modalities, three clinical applications are presented for prostate cancer segmentation, including modality availability assessment, importance quantification and rule discovery.