# Diagnose Like a Radiologist: Hybrid Neuro-Probabilistic Reasoning for Attribute-Based Medical Image Diagnosis

arXiv: https://arxiv.org/abs/2208.09282 · HTML: https://mlsanity.com/paper/2208.09282/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 68 references that are in the map.
- First posted on arXiv 19 Aug 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning](https://mlsanity.com/paper/2006.06649/) (2020)
- [Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification](https://mlsanity.com/paper/1801.09927/) (2018)
- [StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing](https://mlsanity.com/paper/1806.07832/) (2018)
- [Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction](https://mlsanity.com/paper/1802.03584/) (2018)
- [Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learning](https://mlsanity.com/paper/1704.08797/) (2017)
- [TumorNet: Lung Nodule Characterization Using Multi-View Convolutional Neural Network with Gaussian Process](https://mlsanity.com/paper/1703.00645/) (2017)
- [Constructing Deep Neural Networks by Bayesian Network Structure Learning](https://mlsanity.com/paper/1806.09141/) (2018)
- [Characterization of Lung Nodule Malignancy using Hybrid Shape and Appearance Features](https://mlsanity.com/paper/1609.06668/) (2016)
- [Learning Bayesian Networks with Low Rank Conditional Probability Tables](https://mlsanity.com/paper/1905.12552/) (2019)

## Built on by (most cited first, 21 of 21)

- [Advancing Radiograph Representation Learning with Masked Record Modeling](https://mlsanity.com/paper/2301.13155/) (2023)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](https://mlsanity.com/paper/2306.00864/) (2023)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization](https://mlsanity.com/paper/2210.07571/) (2022)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [Deep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions](https://mlsanity.com/paper/2304.06662/) (2023)
- [Attention-Map Augmentation for Hypercomplex Breast Cancer Classification](https://mlsanity.com/paper/2310.07633/) (2023)
- [Generalizing Medical Image Representations via Quaternion Wavelet Networks](https://mlsanity.com/paper/2310.10224/) (2023)
- [Leveraging Anatomical Constraints with Uncertainty for Pneumothorax Segmentation](https://mlsanity.com/paper/2311.15213/) (2023)
- [Two-dimensional Taxonomy for N-ary Knowledge Representation Learning Methods](https://mlsanity.com/paper/2506.05626/) (2025)
- [Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2602.21657/) (2026)
- [ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training](https://mlsanity.com/paper/2606.00602/) (2026)
- [Learning Discretized Bayesian Networks with GOMEA](https://mlsanity.com/paper/2402.12175/) (2024)
- [ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features](https://mlsanity.com/paper/2409.15744/) (2024)
- [Longitudinal Causal Image Synthesis](https://mlsanity.com/paper/2410.17691/) (2024)
- [Adversarial Purification by Consistency-aware Latent Space Optimization on Data Manifolds](https://mlsanity.com/paper/2412.08394/) (2024)
- [Expertise-aware Multi-LLM Recruitment and Collaboration for Medical Decision-Making](https://mlsanity.com/paper/2508.13754/) (2025)
- [SP-Det: Self-Prompted Dual-Text Fusion for Generalized Multi-Label Lesion Detection](https://mlsanity.com/paper/2512.04875/) (2025)
- [Harnessing Lightweight Transformer with Contextual Synergic Enhancement for Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.23390/) (2026)
- [Vol-Mark: A Watermark for 3D Medical Volume Data Via Cubic Difference Expansion and Contrastive Learning](https://mlsanity.com/paper/2605.04705/) (2026)

## Abstract (arXiv)

During clinical practice, radiologists often use attributes, e.g. morphological and appearance characteristics of a lesion, to aid disease diagnosis. Effectively modeling attributes as well as all relationships involving attributes could boost the generalization ability and verifiability of medical image diagnosis algorithms. In this paper, we introduce a hybrid neuro-probabilistic reasoning algorithm for verifiable attribute-based medical image diagnosis. There are two parallel branches in our hybrid algorithm, a Bayesian network branch performing probabilistic causal relationship reasoning and a graph convolutional network branch performing more generic relational modeling and reasoning using a feature representation. Tight coupling between these two branches is achieved via a cross-network attention mechanism and the fusion of their classification results. We have successfully applied our hybrid reasoning algorithm to two challenging medical image diagnosis tasks. On the LIDC-IDRI benchmark dataset for benign-malignant classification of pulmonary nodules in CT images, our method achieves a new state-of-the-art accuracy of 95.36% and an AUC of 96.54%. Our method also achieves a 3.24% accuracy improvement on an in-house chest X-ray image dataset for tuberculosis diagnosis. Our ablation study indicates that our hybrid algorithm achieves a much better generalization performance than a pure neural network architecture under very limited training data.