mlsanityData to 7 Oct 2026

Large-Language-Model Empowered Dose Volume Histogram Prediction for Intensity Modulated Radiotherapy

arXiv 2402.07167 ↗First posted 11 Feb 2024cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Treatment planning is currently a patient specific, time-consuming, and resource demanding task in radiotherapy. Dose-volume histogram (DVH) prediction plays a critical role in automating this process. The geometric relationship between DVHs in radiotherapy plans and organs-at-risk (OAR) and planning target volume (PTV) has been well established. This study explores the potential of deep learning models for predicting DVHs using images and subsequent human intervention facilitated by a large-language model (LLM) to enhance the planning quality. We propose a pipeline to convert unstructured images to a structured graph consisting of image-patch nodes and dose nodes. A novel Dose Graph Neural Network (DoseGNN) model is developed for predicting DVHs from the structured graph. The proposed DoseGNN is enhanced with the LLM to encode massive knowledge from prescriptions and interactive instructions from clinicians. In this study, we introduced an online human-AI collaboration (OHAC) system as a practical implementation of the concept proposed for the automation of intensity-modulated radiotherapy (IMRT) planning. In comparison to the widely-employed DL models used in radiotherapy, DoseGNN achieved mean square errors that were 80%, 76% and 41.0% of those predicted by Swin U-Net Transformer, 3D U-Net CNN and vanilla MLP, respectively. Moreover, the LLM-empowered DoseGNN model facilitates seamless adjustment to treatment plans through interaction with clinicians using natural language.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 65 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

37 of 65 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2017Graph Attention Networkscited by 8,854
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Inductive Representation Learning on Large Graphscited by 6,284
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
25 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2018How Powerful are Graph Neural Networks?cited by 4,355
  3. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  8. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,097
  9. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  10. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  11. 2018GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Modelscited by 406
  12. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  13. 2015Diffusion-Convolutional Neural Networkscited by 268
  14. 2016Volumetric and Multi-View CNNs for Object Classification on 3D Datacited by 240
  15. 2020U-Net and its variants for medical image segmentation: theory and applicationscited by 169
  16. 2017Predicting multicellular function through multi-layer tissue networkscited by 138
  17. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  18. 2021Nested Graph Neural Networkscited by 136
  19. 2020Handling Missing Data with Graph Representation Learningcited by 132
  20. 2020Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?cited by 58
  21. 2023CktGNN: Circuit Graph Neural Network for Electronic Design Automationcited by 55
  22. 2018Graph Capsule Convolutional Neural Networkscited by 27
  23. 2016FPNN: Field Probing Neural Networks for 3D Datacited by 25
  24. 2022PACE: A Parallelizable Computation Encoder for Directed Acyclic Graphscited by 20
  25. 2024DoseGNN: Improving the Performance of Deep Learning Models in Adaptive Dose-Volume Histogram Prediction through Graph Neural Networkscited by 2

Built on by

Not cited yet.