# Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing

arXiv: https://arxiv.org/abs/2504.01282 · HTML: https://mlsanity.com/paper/2504.01282/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering](https://mlsanity.com/paper/2406.12334/) (2024)
- [Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling](https://mlsanity.com/paper/2408.17017/) (2024)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [Consistency Analysis of ChatGPT](https://mlsanity.com/paper/2303.06273/) (2023)
- [Robustness of Learning from Task Instructions](https://mlsanity.com/paper/2212.03813/) (2022)
- [Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility](https://mlsanity.com/paper/2305.10235/) (2023)
- [AAAR-1.0: Assessing AI's Potential to Assist Research](https://mlsanity.com/paper/2410.22394/) (2024)
- [Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks](https://mlsanity.com/paper/2503.16974/) (2025)
- [One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations](https://mlsanity.com/paper/2405.05581/) (2024)
- [Order-Independence Without Fine Tuning](https://mlsanity.com/paper/2406.06581/) (2024)

## Built on by (most cited first, 14 of 14)

- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?](https://mlsanity.com/paper/2506.00751/) (2025)
- [JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems](https://mlsanity.com/paper/2604.23478/) (2026)
- [MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering](https://mlsanity.com/paper/2505.24040/) (2025)
- [Losing the Plot: How VLM responses degrade on imperfect charts](https://mlsanity.com/paper/2509.18425/) (2025)
- [Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoning](https://mlsanity.com/paper/2603.09512/) (2026)
- [Internal Planning in Language Models: Characterizing Horizon and Branch Awareness](https://mlsanity.com/paper/2509.25260/) (2025)
- [VISTA: Knowledge-Driven Vessel Trajectory Imputation with Repair Provenance](https://mlsanity.com/paper/2601.06940/) (2026)
- [Bridging the Semantic Gaps: Improving Medical VQA Consistency with LLM-Augmented Question Sets](https://mlsanity.com/paper/2504.11777/) (2025)
- [Precision in Practice: Knowledge Guided Code Summarizing Grounded in Industrial Expectations](https://mlsanity.com/paper/2602.03400/) (2026)
- [Understanding Degradation with Vision Language Model](https://mlsanity.com/paper/2602.04565/) (2026)
- [Strengthening Human-Centric Chain-of-Thought Reasoning Integrity in LLMs via a Structured Prompt Framework](https://mlsanity.com/paper/2604.04852/) (2026)
- [PEMAND: Persona-Enriched Multi-Agent Negotiation for Household Decision-Making](https://mlsanity.com/paper/2604.10475/) (2026)
- [Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification](https://mlsanity.com/paper/2607.16868/) (2026)

## Abstract (arXiv)

While the inconsistency of LLMs is not a novel topic, prior research has predominantly addressed two types of generative inconsistencies: i) Randomness Inconsistency: running the same LLM multiple trials, yielding varying responses; ii) Paraphrase Inconsistency: paraphrased prompts result in different responses from the same LLM. Randomness Inconsistency arises from the inherent randomness due to stochastic sampling in generative models, while Paraphrase Inconsistency is a consequence of the language modeling objectives, where paraphrased prompts alter the distribution of vocabulary logits. This research discovers Prompt-Reverse Inconsistency (PRIN), a new form of LLM self-inconsistency: given a question and a couple of LLM-generated answer candidates, the LLM often has conflicting responses when prompted "Which are correct answers?" and "Which are incorrect answers?". PRIN poses a big concern as it undermines the credibility of LLM-as-a-judge, and suggests a challenge for LLMs to adhere to basic logical rules. We conduct a series of experiments to investigate PRIN, examining the extent of PRIN across different LLMs, methods to mitigate it, potential applications, and its relationship with Randomness Inconsistency and Paraphrase Inconsistency. As the first study to explore PRIN, our findings offer valuable insights into the inner workings of LLMs and contribute to advancing trustworthy AI.