mlsanityData to 7 Oct 2026

POE: Process of Elimination for Multiple Choice Reasoning

arXiv 2310.15575 ↗First posted 24 Oct 2023cs.CLcited by 7

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 25 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

22 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Prompting16 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales3 Nov

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
7 more
  1. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  2. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  3. 2020Event Extraction by Answering (Almost) Natural Questionscited by 177
  4. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  5. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  6. 2023Mitigating Label Biases for In-context Learningcited by 88
  7. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 3.

7 papers.

  1. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  2. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  3. 2024MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Modelscited by 5
  4. 2026Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Spacecited by 2
  5. 2025Option-ID Based Elimination For Multiple Choice Questionscited by 1
  6. 2026Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluationcited by 0
  7. 2026StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning
  2. 2025Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers
  3. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
  4. 2025Reasoning Models are Test Exploiters: Rethinking Multiple-Choice
  5. 2022Towards Reasoning in Large Language Models: A Survey
  6. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
  7. 2024Multi-Step Reasoning with Large Language Models, a Survey
  8. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches