# POE: Process of Elimination for Multiple Choice Reasoning

arXiv: https://arxiv.org/abs/2310.15575 · HTML: https://mlsanity.com/paper/2310.15575/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [SatLM: Satisfiability-Aided Language Models Using Declarative Prompting](https://mlsanity.com/paper/2305.09656/) (2023)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales](https://mlsanity.com/paper/2211.01562/) (2022)
- [Coherence boosting: When your pretrained language model is not paying enough attention](https://mlsanity.com/paper/2110.08294/) (2021)

## Built on by (most cited first, 8 of 8)

- [Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?](https://mlsanity.com/paper/2402.12483/) (2024)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Models](https://mlsanity.com/paper/2412.07148/) (2024)
- [Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space](https://mlsanity.com/paper/2604.04944/) (2026)
- [Option-ID Based Elimination For Multiple Choice Questions](https://mlsanity.com/paper/2501.15175/) (2025)
- [Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation](https://mlsanity.com/paper/2608.29887/) (2026)
- [StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions](https://mlsanity.com/paper/2609.01081/) (2026)
- [Consider, Then Choose: Order-Invariant Decisions in System One Models](https://mlsanity.com/paper/2610.18233/) (2026)

## Abstract (arXiv)

Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.