mlsanityData to 7 Oct 2026

LViT: Language meets Vision Transformer in Medical Image Segmentation

arXiv 2206.14718 ↗First posted 29 Jun 2022cs.CVcited by 110

Not on the map. Why?

Abstract (arXiv)

Deep learning has been widely used in medical image segmentation and other aspects. However, the performance of existing medical image segmentation models has been limited by the challenge of obtaining sufficient high-quality labeled data due to the prohibitive data annotation cost. To alleviate this limitation, we propose a new text-augmented medical image segmentation model LViT (Language meets Vision Transformer). In our LViT model, medical text annotation is incorporated to compensate for the quality deficiency in image data. In addition, the text information can guide to generate pseudo labels of improved quality in the semi-supervised learning. We also propose an Exponential Pseudo label Iteration mechanism (EPI) to help the Pixel-Level Attention Module (PLAM) preserve local image features in semi-supervised LViT setting. In our model, LV (Language-Vision) loss is designed to supervise the training of unlabeled images using text information directly. For evaluation, we construct three multimodal medical segmentation datasets (image + text) containing X-rays and CT images. Experimental results show that our proposed LViT has superior segmentation performance in both fully-supervised and semi-supervised setting. The code and datasets are available at https://github.com/HUANGLIZI/LViT.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 29 Jun 2022.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  11. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  12. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
31 more
  1. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  5. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  6. 2017Residual Attention Network for Image Classificationcited by 385
  7. 2019Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentationcited by 378
  8. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  9. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 319
  10. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 282
  11. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  12. 2020Semi-supervised Medical Image Segmentation through Dual-task Consistencycited by 215
  13. 2021UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformercited by 197
  14. 2021Mutual Consistency Learning for Semi-supervised Medical Image Segmentationcited by 107
  15. 2020MosMedData: Chest CT Scans With COVID-19 Related Findings Datasetcited by 103
  16. 2020Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptationcited by 81
  17. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  18. 2020Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Modelcited by 59
  19. 2022TGANet: Text-guided attention for improved polyp segmentationcited by 59
  20. 2022OSegNet: Operational Segmentation Network for COVID-19 Detection using Chest X-ray Imagescited by 47
  21. 2021Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentationcited by 43
  22. 2021Boundary-aware Transformers for Skin Lesion Segmentationcited by 39
  23. 2021Dual-Consistency Semi-Supervised Learning with Uncertainty Quantification for COVID-19 Lesion Segmentation from CT Imagescited by 22
  24. 2021Improving Pneumonia Localization via Cross-Attention on Medical Images and Reportscited by 21
  25. 2017Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodulescited by 20
  26. 2021GT U-Net: A U-Net Like Group Transformer Network for Tooth Root Segmentationcited by 20
  27. 2020Lymph Node Gross Tumor Volume Detection and Segmentation via Distance-based Gating using 3D CT/PET Imaging in Radiotherapycited by 15
  28. 2021AGMB-Transformer: Anatomy-Guided Multi-Branch Transformer Network for Automated Evaluation of Root Canal Therapycited by 10
  29. 2022Scaling up Multi-domain Semantic Segmentation with Sentence Embeddingscited by 6
  30. 2022Semi-WTC: A Practical Semi-supervised Framework for Attack Categorization through Weight-Task Consistencycited by 6
  31. 2021Self-supervised Tumor Segmentation through Layer Decompositioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 29; 2025: 34; 2026: 31.

110 papers.

Sort:

By area:

  1. 2024Large Language Models for Medicine: A Surveycited by 38
  2. 2023SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentationcited by 34
  3. 2023Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray imagescited by 33
  4. 2023All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuningcited by 32
  5. 2024BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrievalcited by 28
  6. 2024Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentationcited by 27
  7. 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 25
  8. 2024ScribFormer: Transformer Makes CNN Work Better for Scribble-based Medical Image Segmentationcited by 24
  9. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  10. 2023ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embeddingcited by 23
  11. 2025MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimizationcited by 23
  12. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
38 more
  1. 2024VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledgecited by 19
  2. 2022TFCNs: A CNN-Transformer Hybrid Network for Medical Image Segmentationcited by 18
  3. 2024Rethinking Abdominal Organ Segmentation (RAOS) in the clinical scenario: A robustness evaluation benchmark with challenging casescited by 18
  4. 2025CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Interventioncited by 15
  5. 2024A Refer-and-Ground Multimodal Large Language Model for Biomedicinecited by 14
  6. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  7. 2024Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Modelscited by 13
  8. 2023nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performancecited by 12
  9. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  10. 2025CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentationcited by 12
  11. 2024SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidancecited by 11
  12. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  13. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  14. 2023ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothoraxcited by 10
  15. 2023Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentationcited by 10
  16. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  17. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
  18. 2023Coarse-to-Fine Covid-19 Segmentation via Vision-Language Alignmentcited by 7
  19. 2024Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignmentcited by 7
  20. 2025Towards Universal Text-driven CT Image Segmentationcited by 7
  21. 2023Data efficient deep learning for medical image analysis: A surveycited by 6
  22. 2024Denoising Plane Wave Ultrasound Images Using Diffusion Probabilistic Modelscited by 6
  23. 2024Curriculum Prompting Foundation Models for Medical Image Segmentationcited by 6
  24. 2024Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignmentscited by 6
  25. 2025Applications of Large Models in Medicinecited by 6
  26. 2025STPNet: Scale-aware Text Prompt Network for Medical Image Segmentationcited by 6
  27. 2025Alleviating Textual Reliance in Medical Language-guided Segmentation via Prototype-driven Semantic Approximationcited by 6
  28. 2024NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentationcited by 5
  29. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  30. 2025From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentationcited by 5
  31. 2025Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Datasetcited by 5
  32. 2023A Survey on Image-text Multimodal Modelscited by 4
  33. 2024MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentationcited by 4
  34. 2025TextBraTS: Text-Guided Volumetric Brain Tumor Segmentation with Innovative Dataset Development and Fusion Module Explorationcited by 4
  35. 2025Frequency-domain Multi-modal Fusion for Language-guided Medical Image Segmentationcited by 4
  36. 2023Multiscale Progressive Text Prompt Network for Medical Image Segmentationcited by 3
  37. 2026Uncertainty-Aware Vision-Language Segmentation for Medical Imagingcited by 3
  38. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentationcited by 3

50 most cited of 110 All 110 (JSON)