mlsanityData to 7 Oct 2026

Surface Form Competition: Why the Highest Probability Answer Isn't Always Right

arXiv 2104.08315 ↗First posted 16 Apr 2021cs.CLcited by 256

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have shown promising results in zero-shot settings (Brown et al.,2020; Radford et al., 2019). For example, they can perform multiple choice tasks simply by conditioning on a question and selecting the answer with the highest probability. However, ranking by string probability can be problematic due to surface form competition-wherein different surface forms compete for probability mass, even if they represent the same underlying concept, e.g. "computer" and "PC." Since probability mass is finite, this lowers the probability of the correct answer, due to competition from other strings that are valid answers (but not one of the multiple choice options). We introduce Domain Conditional Pointwise Mutual Information, an alternative scoring function that directly compensates for surface form competition by simply reweighing each option according to a term that is proportional to its a priori likelihood within the context of the specific zero-shot task. It achieves consistent gains in zero-shot performance over both calibrated (Zhao et al., 2021) and uncalibrated scoring functions on all GPT-2 and GPT-3 models over a variety of multiple choice datasets.

Key facts. Cited by 256 papers in the map (data through 7 Oct 2026). Builds on 30 of its 65 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

30 of 65 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
18 more
  1. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  6. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  7. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  8. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  9. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  10. 2019Sparse Sequence-to-Sequence Modelscited by 167
  11. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  12. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  13. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  14. 2019Target-Guided Open-Domain Conversationcited by 66
  15. 2016Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversationcited by 35
  16. 2018Modeling Semantic Plausibility by Injecting World Knowledgecited by 24
  17. 2019Improving Neural Story Generation by Targeted Common Sense Groundingcited by 16
  18. 2019Unsupervised Context Rewriting for Open Domain Conversationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 44; 2023: 63; 2024: 48; 2025: 36; 2026: 53.

256 papers.

Sort:

By area:

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  11. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  12. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
38 more
  1. 2022Can language models learn from explanations in context?cited by 248
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  3. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  4. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  5. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  6. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  7. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  8. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  9. 2022Active Example Selection for In-Context Learningcited by 195
  10. 2024Scaling up Masked Diffusion Models on Textcited by 195
  11. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  12. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  13. 2022Language models show human-like content effects on reasoning taskscited by 171
  14. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  15. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  16. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  17. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  18. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  19. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  20. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  21. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  22. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  23. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  24. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  25. 2023Mitigating Label Biases for In-context Learningcited by 88
  26. 2024Tuning Language Models by Proxycited by 88
  27. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  28. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  29. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  30. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  31. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  32. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  33. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  34. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  35. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  36. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  37. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
  38. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64

50 most cited of 256 All 256 (JSON)