mlsanityData to 7 Oct 2026

Noisy Channel Language Model Prompting for Few-Shot Text Classification

arXiv 2108.04106 ↗First posted 9 Aug 2021cs.CL cs.AIcited by 163

Not on the map. Why?

Abstract (arXiv)

We introduce a noisy channel approach for language model prompting in few-shot text classification. Instead of computing the likelihood of the label given the input (referred as direct models), channel models compute the conditional probability of the input given the label, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context demonstration or prompt tuning. Our experiments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their stability, i.e., lower variance and higher worst-case accuracy. We also present extensive ablations that provide recommendations for when to use channel prompt tuning instead of other competitive methods (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or generalization to unseen labels is required.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). Builds on 32 of its 67 references that are in the map. First posted on arXiv 9 Aug 2021.

Builds on

32 of 67 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
20 more
  1. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2021GPT Understands, Toocited by 715
  4. 2017Learning multiple visual domains with residual adapterscited by 553
  5. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  6. 2021True Few-Shot Learning with Language Modelscited by 343
  7. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  8. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  9. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  10. 2021How Many Data Points is a Prompt Worth?cited by 197
  11. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  12. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  13. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  14. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  15. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  16. 2019Simple and Effective Noisy Channel Modeling for Neural Machine Translationcited by 37
  17. 2018Natural Language to Structured Query Generation via Meta-Learningcited by 33
  18. 2017Generative and Discriminative Text Classification with Recurrent Neural Networkscited by 30
  19. 2016The Neural Noisy Channelcited by 19
  20. 2019Latent-Variable Generative Models for Data-Efficient Text Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 43; 2023: 64; 2024: 35; 2025: 9; 2026: 4.

163 papers.

Sort:

By area:

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2022A Survey on In-context Learningcited by 717
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  7. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
38 more
  1. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  2. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  3. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  4. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  5. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  6. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  7. 2022Natural Language to Code Translation with Executioncited by 122
  8. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  9. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  10. 2023Finding Support Examples for In-Context Learningcited by 104
  11. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  12. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  13. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  14. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  15. 2022Coder Reviewer Reranking for Code Generationcited by 83
  16. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  17. 2022Parallel Context Windows for Large Language Modelscited by 81
  18. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  19. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  20. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  21. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  22. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  23. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  24. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  25. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  26. 2023In-context Example Selection with Influencescited by 60
  27. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 59
  28. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  29. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  30. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 52
  31. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  32. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 48
  33. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  34. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  35. 2023Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractorscited by 44
  36. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  37. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  38. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41

50 most cited of 163 All 163 (JSON)