mlsanityData to 7 Oct 2026

VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis

arXiv 2410.08397 ↗First posted 10 Oct 2024eess.IV cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

We present VoxelPrompt, an end-to-end image analysis agent that tackles free-form radiological tasks. Given any number of volumetric medical images and a natural language prompt, VoxelPrompt integrates a language model that generates executable code to invoke a jointly-trained, adaptable vision network. This code further carries out analytical steps to address practical quantitative aims, such as measuring the growth of a tumor across visits. The pipelines generated by VoxelPrompt automate analyses that currently require practitioners to painstakingly combine multiple specialized vision and statistical tools. We evaluate VoxelPrompt using diverse neuroimaging tasks and show that it can delineate hundreds of anatomical and pathological features, measure complex morphological properties, and perform open-language analysis of lesion characteristics. VoxelPrompt performs these objectives with an accuracy similar to that of specialist single-task models for image analysis, while facilitating a broad range of compositional biomedical workflows.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 48 of its 81 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

48 of 81 references.

References · 48

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  11. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  12. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 507
36 more
  1. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  2. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  3. 2021MetaICL: Learning to Learn In Contextcited by 450
  4. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  5. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  6. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  7. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  8. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  9. 2021SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retrainingcited by 235
  10. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  11. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  12. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  13. 2022SynthStrip: Skull-Stripping for Any Brain Imagecited by 126
  14. 2019Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challengecited by 115
  15. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 111
  16. 2019FastSurfer -- A fast and accurate deep learning based neuroimaging pipelinecited by 110
  17. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  18. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 107
  19. 2022ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation datasetcited by 104
  20. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  21. 2019'Squeeze & Excite' Guided Few-Shot Segmentation of Volumetric Imagescited by 69
  22. 2023Large-Vocabulary Segmentation for Medical Images with Text Promptscited by 60
  23. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
  24. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  25. 2023Modular Visual Question Answering via Code Generationcited by 47
  26. 2021MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learningcited by 45
  27. 2022One Model is All You Need: Multi-Task Learning Enables Simultaneous Histology Image Segmentation and Classificationcited by 38
  28. 2019Intracranial Hemorrhage Segmentation Using Deep Convolutional Modelcited by 29
  29. 2023BHSD: A 3D Multi-Class Brain Hemorrhage Segmentation Datasetcited by 27
  30. 2023Neuralizer: General Neuroimage Analysis without Re-Trainingcited by 24
  31. 2024Tyche: Stochastic In-Context Learning for Medical Image Segmentationcited by 20
  32. 2023Anatomy-aware and acquisition-agnostic joint registration with SynthMorphcited by 19
  33. 2024A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Expertscited by 13
  34. 2020Extending Unsupervised Neural Image Compression With Supervised Multitask Learningcited by 12
  35. 2021Joint Registration and Segmentation via Multi-Task Learning for Adaptive Radiotherapy of Prostate Cancercited by 8
  36. 2020Simulation of Brain Resection for Cavity Segmentation Using Self-Supervised and Semi-Supervised Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflowscited by 5
  2. 2026GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRIcited by 1
  3. 2026textttDisMorph: learning to disentangle technical distortions from true biological changecited by 0
  4. 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0
  5. 2026A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scanscited by 0
  6. 2026Towards Fully Automated Medical Imaging Code Generation via Validation-based Context Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imaging
  2. 2026Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysis
  3. 2026MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Models
  4. 2025Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation
  5. 2025VoxTell: Free-Text Promptable Universal 3D Medical Image Segmentation
  6. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks
  7. 2026RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding
  8. 2025MedVision: Benchmarking Quantitative Medical Image Analysis