# Task-Based Assessment for Neural Networks: Evaluating Undersampled MRI Reconstructions based on Human Observer Signal Detection

arXiv: https://arxiv.org/abs/2210.12161 · HTML: https://mlsanity.com/paper/2210.12161/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 9 of 9)

- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [Learning a Variational Network for Reconstruction of Accelerated MRI Data](https://mlsanity.com/paper/1704.00447/) (2017)
- [On hallucinations in tomographic image reconstruction](https://mlsanity.com/paper/2012.00646/) (2020)
- [Image Reconstruction: From Sparsity to Data-adaptive Methods and Machine Learning](https://mlsanity.com/paper/1904.02816/) (2019)
- [Approximating the Ideal Observer for joint signal detection and localization tasks by use of supervised learning methods](https://mlsanity.com/paper/2006.00112/) (2020)
- [Impact of deep learning-based image super-resolution on binary signal detection](https://mlsanity.com/paper/2107.02338/) (2021)
- [Artifact- and content-specific quality assessment for MRI with image rulers](https://mlsanity.com/paper/2111.03780/) (2021)
- [Investigating the limited performance of a deep-learning-based SPECT denoising approach: An observer-study-based characterization](https://mlsanity.com/paper/2203.01918/) (2022)

## Abstract (arXiv)

Recent research has explored using neural networks to reconstruct undersampled magnetic resonance imaging (MRI) data. Because of the complexity of the artifacts in the reconstructed images, there is a need to develop task-based approaches of image quality. Common metrics for evaluating image quality like the normalized root mean squared error (NRMSE) and structural similarity (SSIM) are global metrics which average out impact of subtle features in the images. Using measures of image quality which incorporate a subtle signal for a specific task allow for image quality assessment which locally evaluates the effect of undersampling on a signal. We used a U-Net to reconstruct under-sampled images with 2x, 3x, 4x and 5x fold 1-D undersampling rates. Cross validation was performed for a 500 and a 4000 image training set with both structural similarity (SSIM) and mean squared error (MSE) losses. A two alternative forced choice (2-AFC) observer study was carried out for detecting a subtle signal (small blurred disk) from images with the 4000 image training set. We found that for both loss functions and training set sizes, the human observer performance on the 2-AFC studies led to a choice of a 2x undersampling but the SSIM and NRMSE led to a choice of a 3x undersampling. For this task, SSIM and NRMSE led to an overestimate of the achievable undersampling using a U-Net before a steep loss of image quality when compared to the performance of human observers in the detection of a subtle lesion.