# CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation

arXiv: https://arxiv.org/abs/2309.03320 · HTML: https://mlsanity.com/paper/2309.03320/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 63 references that are in the map.
- First posted on arXiv 6 Sep 2023.

## Builds on (most cited first, 30 of 43)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions](https://mlsanity.com/paper/2003.01826/) (2020)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up](https://mlsanity.com/paper/2102.07074/) (2021)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Image Generators with Conditionally-Independent Pixel Synthesis](https://mlsanity.com/paper/2011.13775/) (2020)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Implicit Neural Representation in Medical Imaging: A Comparative Survey](https://mlsanity.com/paper/2307.16142/) (2023)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [Reconstructing continuous distributions of 3D protein structure from cryo-EM images](https://mlsanity.com/paper/1909.05215/) (2019)
- [Spatially-Adaptive Pixelwise Networks for Fast Image Translation](https://mlsanity.com/paper/2012.02992/) (2020)

## Built on by (most cited first, 3 of 3)

- [Vestibular schwannoma growth prediction from longitudinal MRI by time conditioned neural fields](https://mlsanity.com/paper/2404.02614/) (2024)
- [A deep learning model to reduce agent dose for contrast-enhanced MRI of the cerebellopontine angle cistern](https://mlsanity.com/paper/2511.20926/) (2025)
- [Implicit Neural Representations for Multimodal Longitudinal Image Imputation and Interpolation](https://mlsanity.com/paper/2608.02324/) (2026)

## Abstract (arXiv)

Multi-sequence magnetic resonance imaging (MRI) has found wide applications in both modern clinical studies and deep learning research. However, in clinical practice, it frequently occurs that one or more of the MRI sequences are missing due to different image acquisition protocols or contrast agent contraindications of patients, limiting the utilization of deep learning models trained on multi-sequence data. One promising approach is to leverage generative models to synthesize the missing sequences, which can serve as a surrogate acquisition. State-of-the-art methods tackling this problem are based on convolutional neural networks (CNN) which usually suffer from spectral biases, resulting in poor reconstruction of high-frequency fine details. In this paper, we propose Conditional Neural fields with Shift modulation (CoNeS), a model that takes voxel coordinates as input and learns a representation of the target images for multi-sequence MRI translation. The proposed model uses a multi-layer perceptron (MLP) instead of a CNN as the decoder for pixel-to-pixel mapping. Hence, each target image is represented as a neural field that is conditioned on the source image via shift modulation with a learned latent code. Experiments on BraTS 2018 and an in-house clinical dataset of vestibular schwannoma patients showed that the proposed method outperformed state-of-the-art methods for multi-sequence MRI translation both visually and quantitatively. Moreover, we conducted spectral analysis, showing that CoNeS was able to overcome the spectral bias issue common in conventional CNN models. To further evaluate the usage of synthesized images in clinical downstream tasks, we tested a segmentation network using the synthesized images at inference.