mlsanityData to 14 Oct 2026

Pre-Trained LLM is a Semantic-Aware and Generalizable Segmentation Booster

arXiv 2506.18034 ↗First posted 22 Jun 2025cs.CV cs.AI cs.MMcited by 7

Not on the map. Why?

Abstract (arXiv)

With the advancement of Large Language Model (LLM) for natural language processing, this paper presents an intriguing finding: a frozen pre-trained LLM layer can process visual tokens for medical image segmentation tasks. Specifically, we propose a simple hybrid structure that integrates a pre-trained, frozen LLM layer within the CNN encoder-decoder segmentation framework (LLM4Seg). Surprisingly, this design improves segmentation performance with a minimal increase in trainable parameters across various modalities, including ultrasound, dermoscopy, polypscopy, and CT scans. Our in-depth analysis reveals the potential of transferring LLM's semantic awareness to enhance segmentation tasks, offering both improved global understanding and better local modeling capabilities. The improvement proves robust across different LLMs, validated using LLaMA and DeepSeek.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 22 Jun 2025.

Builds on

28 of 39 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  9. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  10. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  11. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 669
  12. 2019UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentationcited by 586
16 more
  1. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  2. 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
  3. 2021UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformercited by 197
  4. 2022UNeXt: MLP-based Rapid Medical Image Segmentation Networkcited by 185
  5. 2023MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentationcited by 162
  6. 2021DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentationcited by 126
  7. 20223D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentationcited by 94
  8. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  9. 2022Investigating Why Contrastive Learning Benefits Robustness Against Label Noisecited by 46
  10. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  11. 2023CMUNeXt: An Efficient Medical Image Segmentation Network based on Large Kernel and Skip Fusioncited by 31
  12. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  13. 2024HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Trainingcited by 12
  14. 2025Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentationcited by 12
  15. 2025Towards Accurate Unified Anomaly Segmentationcited by 7
  16. 2023MobileUtr: Revisiting the relationship between light-weight CNN and Transformer for efficient medical image segmentationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Mobile U-ViT: Revisiting large kernel and U-shaped ViT for efficient medical image segmentationcited by 6
  2. 2025Large Language Model Evaluated Stand-alone Attention-Assisted Graph Neural Network with Spatial and Structural Information Interaction for Precise Endoscopic Image Segmentationcited by 0
  3. 2025Externally Validated Breast Ultrasound Segmentation via Multi-task Learning with BI-RADS-Consistent Morphological Priorscited by 0
  4. 2026UCAD: Uncertainty-guided Contour-aware Displacement for semi-supervised medical image segmentationcited by 0
  5. 2026Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentationcited by 0
  6. 2026DiffAttn: Diffusion-Based Drivers' Visual Attention Prediction with LLM-Enhanced Semantic Reasoningcited by 0
  7. 2026APRIL-MedSeg: A Modular Medical Image Segmentation Toolbox Embracing Modern Paradigmscited by 0