mlsanityData to 14 Oct 2026

Residual-based Language Models are Free Boosters for Biomedical Imaging

arXiv 2403.17343 ↗First posted 26 Mar 2024cs.CV cs.CL cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

In this study, we uncover the unexpected efficacy of residual-based large language models (LLMs) as part of encoders for biomedical imaging tasks, a domain traditionally devoid of language or textual data. The approach diverges from established methodologies by utilizing a frozen transformer block, extracted from pre-trained LLMs, as an innovative encoder layer for the direct processing of visual tokens. This strategy represents a significant departure from the standard multi-modal vision-language frameworks, which typically hinge on language-driven prompts and inputs. We found that these LLMs could boost performance across a spectrum of biomedical imaging applications, including both 2D and 3D visual classification tasks, serving as plug-and-play boosters. More interestingly, as a byproduct, we found that the proposed framework achieved superior performance, setting new state-of-the-art results on extensive, standardized datasets in MedMNIST-2D and 3D. Through this work, we aim to open new avenues for employing LLMs in biomedical imaging and enriching the understanding of their potential in this specialized domain.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 54 of its 90 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

54 of 90 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  13. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  14. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  17. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  18. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 320
  19. 2022Linearly Mapping from Image to Text Spacecited by 142
  20. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  21. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  22. 2021UTNet: A Hybrid Transformer Architecture for Medical Image Segmentationcited by 116
  23. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  24. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  25. 2021TransMed: Transformers Advance Multi-modal Medical Image Classificationcited by 86
  26. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  27. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 38
  28. 2023When do you need Chain-of-Thought Prompting for ChatGPT?cited by 35
  29. 2023Improving Depth Gradient Continuity in Transformers: A Comparative Study on Monocular Depth Estimation with CNNcited by 35
  30. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 31
  31. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 27
  32. 2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
  33. 2023ReConTab: Regularized Contrastive Representation Learning for Tabular Datacited by 25
  34. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  35. 2022Optimizing Nitrogen Management with Deep Reinforcement Learning and Crop Simulationscited by 20
  36. 2022Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrievalcited by 16
  37. 2023Extended Agriculture-Vision: An Extension of a Large Aerial Image Dataset for Agricultural Pattern Analysiscited by 14
  38. 2022Optimizing Crop Management with Reinforcement Learning and Imitation Learningcited by 13

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 3; 2026: 4.

24 papers.

Sort:

By area:

  1. 2024Restoring Images in Adverse Weather Conditions via Histogram Transformercited by 73
  2. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  3. 2024MemoryMamba: Memory-Augmented State Space Model for Defect Recognitioncited by 24
  4. 2024Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approachcited by 24
  5. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  6. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationalecited by 14
  7. 2024The New Agronomists: Language Models are Experts in Crop Managementcited by 13
  8. 2024InsectMamba: Insect Pest Classification with State Space Modelcited by 11
  9. 2024Research on the Application of Computer Vision Based on Deep Learning in Autonomous Driving Technologycited by 9
  10. 2024A Hybrid Transformer-Mamba Network for Single Image Derainingcited by 6
  11. 2024MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentationcited by 4
  12. 2024Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Modelscited by 4
12 more
  1. 2024Large Language Models for Multimodal Deformable Image Registrationcited by 3
  2. 2024SAM-Guided Masked Token Prediction for 3D Scene Understandingcited by 3
  3. 2024Optimized Coordination Strategy for Multi-Aerospace Systems in Pick-and-Place Tasks By Deep Neural Networkcited by 3
  4. 2026OmniRad: A Radiological Foundation Model for Multi-Task Medical Image Analysiscited by 3
  5. 2024Deep Representation Learning for Multi-functional Degradation Modeling of Community-dwelling Aging Populationcited by 1
  6. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  7. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
  8. 2024CROPS: A Deployable Crop Management System Over All Possible State Availabilitiescited by 0
  9. 2025UniCAD: Efficient and Extendable Architecture for Multi-Task Computer-Aided Diagnosis Systemcited by 0
  10. 2026Axial-Centric Cross-Plane Attention for 3D Medical Image Classificationcited by 0
  11. 2026HierBias: Context-Conditioned Hierarchical Media Bias Detection with Multi-Task Type Classificationcited by 0
  12. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0