HGT: A Hierarchical GCN-Based Transformer for Multimodal Periprosthetic Joint Infection Diagnosis Using CT Images and Text
Not on the map. Why?
Abstract (arXiv)
Prosthetic Joint Infection (PJI) is a prevalent and severe complication characterized by high diagnostic challenges. Currently, a unified diagnostic standard incorporating both computed tomography (CT) images and numerical text data for PJI remains unestablished, owing to the substantial noise in CT images and the disparity in data volume between CT images and text data. This study introduces a diagnostic method, HGT, based on deep learning and multimodal techniques. It effectively merges features from CT scan images and patients' numerical text data via a Unidirectional Selective Attention (USA) mechanism and a graph convolutional network (GCN)-based feature fusion network. We evaluated the proposed method on a custom-built multimodal PJI dataset, assessing its performance through ablation experiments and interpretability evaluations. Our method achieved an accuracy (ACC) of 91.4% and an area under the curve (AUC) of 95.9%, outperforming recent multimodal approaches by 2.9% in ACC and 2.2% in AUC, with a parameter count of only 68M. Notably, the interpretability results highlighted our model's strong focus and localization capabilities at lesion sites. This proposed method could provide clinicians with additional diagnostic tools to enhance accuracy and efficiency in clinical practice.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 53 references that are in the map. First posted on arXiv 29 May 2023.
Builds on
27 of 53 references.
References · 27
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Inductive Representation Learning on Large Graphscited by 6,320
- 2016Layer Normalizationcited by 5,669
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2015VQA: Visual Question Answeringcited by 2,506
- 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2017Decoupled Weight Decay Regularizationcited by 1,831
15 more
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
- 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
- 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
- 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
- 2022MaxViT: Multi-Axis Vision Transformercited by 476
- 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
- 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
- 2019Fine-tune BERT for Extractive Summarizationcited by 122
- 2021TransMed: Transformers Advance Multi-modal Medical Image Classificationcited by 86
- 2018ExFuse: Enhancing Feature Fusion for Semantic Segmentationcited by 42
- 2022NEEDED: Introducing Hierarchical Transformer to Eye Diseases Diagnosiscited by 7
- 2022Automatic Diagnosis of Myocarditis Disease in Cardiac MRI Modality using Deep Transformers and Explainable Artificial Intelligencecited by 2
Built on by
Not cited yet.