mlsanityData to 7 Oct 2026

Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning

arXiv 2301.11916 ↗First posted 27 Jan 2023cs.CL cs.AI cs.LGcited by 149

Not on the map. Why?

Abstract (arXiv)

In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. However, existing literature has highlighted the sensitivity of this capability to the selection of few-shot demonstrations. Current understandings of the underlying mechanisms by which this capability arises from regular language model pretraining objectives remain disconnected from the real-world LLMs. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models. On this premise, we propose an algorithm to select optimal demonstrations from a set of annotated data with a small LM, and then directly generalize the selected demonstrations to larger LMs. We demonstrate significant improvement over baselines, averaged over eight GPT models on eight real-world text classification datasets. We also demonstrate the real-world usefulness of our algorithm on GSM8K, a math word problem dataset. Our empirical findings support our hypothesis that LLMs implicitly infer a latent variable containing task information.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

34 of 52 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
22 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  9. 2022Transformers learn in-context by gradient descentcited by 714
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  12. 2021MetaICL: Learning to Learn In Contextcited by 450
  13. 2021True Few-Shot Learning with Language Modelscited by 343
  14. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  16. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  17. 2015Neural Variational Inference for Text Processingcited by 111
  18. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  19. 2017Discovering Discrete Latent Topics with Neural Variational Inferencecited by 79
  20. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  21. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  22. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 58; 2025: 49; 2026: 22.

149 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2024Many-Shot In-Context Learningcited by 203
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2023In-Context Convergence of Transformerscited by 112
  6. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  7. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  8. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  9. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  10. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  11. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  12. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 63
38 more
  1. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 51
  2. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  3. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  4. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  5. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  6. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  7. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  8. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  9. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  10. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  11. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  12. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  13. 2024MECD: Unlocking Multi-Event Causal Discovery in Video Reasoningcited by 23
  14. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 23
  15. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  16. 2023Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Modelscited by 21
  17. 2023Generative Calibration for In-context Learningcited by 20
  18. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  19. 2024Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundationscited by 19
  20. 2025Enough Coin Flips Can Make LLMs Act Bayesiancited by 19
  21. 2024Assistive Large Language Model Agents for Socially-Aware Negotiation Dialoguescited by 18
  22. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  23. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 17
  24. 2024On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Conceptcited by 16
  25. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  26. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15
  27. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15
  28. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 15
  29. 2024AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modellingcited by 15
  30. 2024RAEmoLLM: Retrieval Augmented LLMs for Cross-Domain Misinformation Detection Using In-Context Learning Based on Emotional Informationcited by 14
  31. 2025The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Modelscited by 14
  32. 2025Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidancecited by 14
  33. 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
  34. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  35. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  36. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  37. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 12
  38. 2024BERTs are Generative In-Context Learnerscited by 11

50 most cited of 149 All 149 (JSON)