# Language Models are Few-shot Learners for Prognostic Prediction

arXiv: https://arxiv.org/abs/2302.12692 · HTML: https://mlsanity.com/paper/2302.12692/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network](https://mlsanity.com/paper/1606.00931/) (2016)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Masked Image Modeling Advances 3D Medical Image Analysis](https://mlsanity.com/paper/2204.11716/) (2022)
- [GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain](https://mlsanity.com/paper/2109.02555/) (2021)

## Built on by (most cited first, 30 of 37)

- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model](https://mlsanity.com/paper/2306.16269/) (2023)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [Zero-Shot Listwise Document Reranking with a Large Language Model](https://mlsanity.com/paper/2305.02156/) (2023)
- [LLMCad: Fast and Scalable On-device Large Language Model Inference](https://mlsanity.com/paper/2309.04255/) (2023)
- [MoT: Memory-of-Thought Enables ChatGPT to Self-Improve](https://mlsanity.com/paper/2305.05181/) (2023)
- [Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting](https://mlsanity.com/paper/2307.01709/) (2023)
- [Prompting Large Language Models for Topic Modeling](https://mlsanity.com/paper/2312.09693/) (2023)
- [ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes](https://mlsanity.com/paper/2403.05795/) (2024)
- [Beyond Prompts: Exploring the Design Space of Mixed-Initiative Co-Creativity Systems](https://mlsanity.com/paper/2305.07465/) (2023)
- [UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?](https://mlsanity.com/paper/2306.16931/) (2023)
- [LLMs-Healthcare : Current Applications and Challenges of Large Language Models in various Medical Specialties](https://mlsanity.com/paper/2311.12882/) (2023)
- [MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation](https://mlsanity.com/paper/2403.14652/) (2024)
- [From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice](https://mlsanity.com/paper/2410.01812/) (2024)
- [AAD-LLM: Adaptive Anomaly Detection Using Large Language Models](https://mlsanity.com/paper/2411.00914/) (2024)
- [Implicit Ray-Transformers for Multi-view Remote Sensing Image Segmentation](https://mlsanity.com/paper/2303.08401/) (2023)
- [Multi-hop Commonsense Knowledge Injection Framework for Zero-Shot Commonsense Question Answering](https://mlsanity.com/paper/2305.05936/) (2023)
- [Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs](https://mlsanity.com/paper/2407.04173/) (2024)
- [CCGen: Explainable Complementary Concept Generation in E-Commerce](https://mlsanity.com/paper/2305.11480/) (2023)
- [PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models](https://mlsanity.com/paper/2312.06093/) (2023)
- [Parameter-Efficient Learning for Text-to-Speech Accent Adaptation](https://mlsanity.com/paper/2305.11320/) (2023)
- [Soft Language Clustering for Multilingual Model Pre-training](https://mlsanity.com/paper/2306.07610/) (2023)
- [RAAD-LLM: Adaptive Anomaly Detection Using LLMs and RAG Integration](https://mlsanity.com/paper/2503.02800/) (2025)
- [Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging](https://mlsanity.com/paper/2506.18434/) (2025)
- [disco: a toolkit for Distributional Control of Generative Models](https://mlsanity.com/paper/2303.05431/) (2023)
- [Understanding Parameter Sharing in Transformers](https://mlsanity.com/paper/2306.09380/) (2023)
- [Ask Language Model to Clean Your Noisy Translation Data](https://mlsanity.com/paper/2310.13469/) (2023)
- [Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension](https://mlsanity.com/paper/2302.13619/) (2023)

## Abstract (arXiv)

Clinical prediction is an essential task in the healthcare industry. However, the recent success of transformers, on which large language models are built, has not been extended to this domain. In this research, we explore the use of transformers and language models in prognostic prediction for immunotherapy using real-world patients' clinical data and molecular profiles. This paper investigates the potential of transformers to improve clinical prediction compared to conventional machine learning approaches and addresses the challenge of few-shot learning in predicting rare disease areas. The study benchmarks the efficacy of baselines and language models on prognostic prediction across multiple cancer types and investigates the impact of different pretrained language models under few-shot regimes. The results demonstrate significant improvements in accuracy and highlight the potential of NLP in clinical research to improve early detection and intervention for different diseases.