mlsanityData to 14 Oct 2026

Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography

arXiv 2403.17834 ↗First posted 26 Mar 2024cs.CVcited by 199

Not on the map. Why?

Cited by 62 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language-image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP's vision encoder with a pretrained large language model, we create CT-CHAT, a vision-language foundational chat model for 3D chest CT volumes. Finetuned on over 2.7 million question-answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP, and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). Builds on 28 of its 73 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

28 of 73 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  7. 2021Scaling Vision Transformerscited by 999
  8. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  9. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2021The Medical Segmentation Decathloncited by 628
  12. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
16 more
  1. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 509
  2. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  3. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  4. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  5. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 255
  6. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 239
  7. 2020A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promisescited by 231
  8. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  9. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  10. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  11. 2020Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumescited by 67
  12. 2023GenerateCT: Text-Conditional Generation of 3D Chest CT Volumescited by 64
  13. 2019Similar Image Search for Histopathology: SMILYcited by 41
  14. 2023MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Imagescited by 41
  15. 2021GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imagingcited by 25
  16. 2023Diffusion-Based Hierarchical Multi-Label Object Detection to Analyze Panoramic Dental X-rayscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 67; 2026: 115.

199 papers.

Sort:

By area:

  1. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understandingcited by 54
  2. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 46
  3. 2026MedGemma 1.5 Technical Reportcited by 45
  4. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 42
  5. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  6. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 30
  7. 2024Learning General-Purpose Biomedical Volume Representations using Randomized Synthesiscited by 29
  8. 2024Large Language Model with Region-guided Referring and Grounding for CT Report Generationcited by 28
  9. 2024An OpenMind for 3D medical vision self-supervised learningcited by 26
  10. 2025FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysiscited by 24
  11. 2025Primus: Enforcing Attention Usage for 3D Medical Image Segmentationcited by 24
  12. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
38 more
  1. 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 21
  2. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imagingcited by 20
  3. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  4. 2025ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reportscited by 17
  5. 20243D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Modelscited by 16
  6. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  7. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  8. 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answeringcited by 14
  9. 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 14
  10. 2025NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRIcited by 13
  11. 2025Vision Language Models in Medicinecited by 12
  12. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 10
  13. 2024Improving Representation of High-frequency Components for Medical Visual Foundation Modelscited by 9
  14. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  15. 2025Revisiting 2D Foundation Models for Scalable 3D Medical Image Classificationcited by 9
  16. 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-trainingcited by 8
  17. 2025CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generationcited by 8
  18. 2026HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agentscited by 8
  19. 2025Towards Universal Text-driven CT Image Segmentationcited by 7
  20. 2025PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomographycited by 7
  21. 2025Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registrationcited by 7
  22. 2025CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generationcited by 6
  23. 2025From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generationcited by 6
  24. 2025Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generationcited by 6
  25. 2025From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Researchcited by 6
  26. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  27. 2026RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomographycited by 6
  28. 2025CTFlow: Video-Inspired Latent Flow Matching for 3D CT Synthesiscited by 5
  29. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  30. 2025Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reportingcited by 5
  31. 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysiscited by 5
  32. 2026U-VLM: Hierarchical Vision Language Modeling for Report Generationcited by 5
  33. 2024ReXplain: Translating Radiology into Patient-Friendly Video Reportscited by 4
  34. 2025RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Miningcited by 4
  35. 2025MedM-VL: What Makes a Good Medical LVLM?cited by 4
  36. 2025Unified Supervision For Vision-Language Modeling in 3D Computed Tomographycited by 4
  37. 2025MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reportingcited by 4
  38. 2025LCMem: A Universal Model for Robust Image Memorization Detectioncited by 4

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
  2. 2025VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
  3. 2026CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling
  4. 2024Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation
  5. 2026Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
  6. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models
  7. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks
  8. 2025Computed Tomography Visual Question Answering with Cross-modal Feature Graphing