# It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning

arXiv: https://arxiv.org/abs/2311.07532 · HTML: https://mlsanity.com/paper/2311.07532/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 3 of its 42 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 3 of 3)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinking](https://mlsanity.com/paper/2310.12342/) (2023)

## Built on by (most cited first, 12 of 12)

- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?](https://mlsanity.com/paper/2410.15512/) (2024)
- [BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks](https://mlsanity.com/paper/2602.06221/) (2026)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [KARL: Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students](https://mlsanity.com/paper/2402.12291/) (2024)
- [Option-ID Based Elimination For Multiple Choice Questions](https://mlsanity.com/paper/2501.15175/) (2025)
- [Can Competition Enhance the Proficiency of Agents Powered by Large Language Models in the Realm of News-driven Time Series Forecasting?](https://mlsanity.com/paper/2504.10210/) (2025)
- [SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation](https://mlsanity.com/paper/2605.12022/) (2026)
- [Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation](https://mlsanity.com/paper/2608.29887/) (2026)
- [StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions](https://mlsanity.com/paper/2609.01081/) (2026)
- [Knowledge-Graph Based Augmentation versus Retrieval Augmented Generation for Cultural-Related Question Answering](https://mlsanity.com/paper/2609.18317/) (2026)

## Abstract (arXiv)

Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work.