mlsanityData to 7 Oct 2026

Sparse Concept Channels in Frozen 3D CT Vision Encoders

arXiv 2607.20993 ↗First posted 23 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large vision-language models are becoming increasingly dominant in 3D medical image interpretation, but we rarely know <i>which</i> internal units encode clinical findings or <i>where</i> that information lives in the representation. We first study this on a 3D chest vision-language model (Pillar-0) by probing its frozen vision embeddings. We show that (i) each radiological finding is encoded by a <i>sparse</i> set of ~10 vision-encoder channels that match full-feature classification performance and far exceed a zero-shot text prompting; (ii) turning off the channels tied to one finding, that finding's score collapses while unrelated labels stay stable; and (iii) the same sparse probe <i>replicates</i> on an architecturally unrelated 3D abdominal VLM (Merlin) suggesting a general property of frozen medical encoders. Our training-free concept channel probe (CCP) method, paired with a corpus-derived report template, outperforms published CT-CHAT on clinical efficacy and NLG metrics (F1 0.549 vs. 0.184; BLEU 0.483 vs. 0.373) at 22x lower latency. Our results provide a clear, reproducible characterization of how frozen medical encoders represent findings, demonstrating direct applicability across models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 8 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

7 of 8 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Comprehensive language-image pre-training for 3D medical image understanding16 Oct

Other references · 6

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  4. 2019Are Sixteen Heads Really Better than One?cited by 803
  5. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  6. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
  2. 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
  3. 2026NV-Reason-CT: 3D Visual Language Model for CT Analysis
  4. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks
  5. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model
  6. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
  7. 2026Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
  8. 2025Unified Supervision For Vision-Language Modeling in 3D Computed Tomography