# DINOv3-Guided Cross Fusion Framework for Semantic-aware CT generation from MRI and CBCT

arXiv: https://arxiv.org/abs/2511.12098 · HTML: https://mlsanity.com/paper/2511.12098/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [Low Dose CT Image Denoising Using a Generative Adversarial Network with Wasserstein Distance and Perceptual Loss](https://mlsanity.com/paper/1708.00961/) (2017)
- [Diffusion Probabilistic Models beat GANs on Medical Images](https://mlsanity.com/paper/2212.07501/) (2022)
- [Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets](https://mlsanity.com/paper/2210.05958/) (2022)
- [Deep learning-based synthetic-CT generation in radiotherapy and PET: a review](https://mlsanity.com/paper/2102.02734/) (2021)
- [SynthRAD2025 Grand Challenge dataset: generating synthetic CTs for radiotherapy](https://mlsanity.com/paper/2502.17609/) (2025)
- [Real-Time Object Detection Meets DINOv3](https://mlsanity.com/paper/2509.20787/) (2025)
- [I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling](https://mlsanity.com/paper/2405.14022/) (2024)
- [SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3](https://mlsanity.com/paper/2509.00833/) (2025)

## Abstract (arXiv)

Generating synthetic CT images from CBCT or MRI has a potential for efficient radiation dose planning and adaptive radiotherapy. However, existing CNN-based models lack global semantic understanding, while Transformers often overfit small medical datasets due to high model capacity and weak inductive bias. To address these limitations, we propose a DINOv3-Guided Cross Fusion (DGCF) framework that integrates a frozen self-supervised DINOv3 Transformer with a trainable CNN encoder-decoder. It hierarchically fuses global representation of Transformer and local features of CNN via a learnable cross fusion module, achieving balanced local appearance and contextual representation. Furthermore, we introduce a Multi-Level DINOv3 Perceptual (MLDP) loss that encourages semantic similarity between synthetic CT and the ground truth in DINOv3's feature space. Experiments on the SynthRAD2023 pelvic dataset demonstrate that DGCF achieved state-of-the-art performance in terms of MS-SSIM, PSNR and segmentation-based metrics on both MRIrightarrowCT and CBCTrightarrowCT translation tasks. To the best of our knowledge, this is the first work to employ DINOv3 representations for medical image translation, highlighting the potential of self-supervised Transformer guidance for semantic-aware CT synthesis. The code is available at https://github.com/HiLab-git/DGCF.