# Conditional and Modal Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2401.17169 · HTML: https://mlsanity.com/paper/2401.17169/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2401.00757/) (2024)
- [Improving Large Language Models in Event Relation Logical Prediction](https://mlsanity.com/paper/2310.09158/) (2023)
- [Are Language Models Worse than Humans at Following Prompts? It's Complicated](https://mlsanity.com/paper/2301.07085/) (2023)

## Built on by (most cited first, 9 of 9)

- [Empowering LLMs with Logical Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2502.15652/) (2025)
- [When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives](https://mlsanity.com/paper/2406.12084/) (2024)
- [Belief in the Machine: Investigating Epistemological Blind Spots of Language Models](https://mlsanity.com/paper/2410.21195/) (2024)
- [Learning Elementary Cellular Automata with Transformers](https://mlsanity.com/paper/2412.01417/) (2024)
- [Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling](https://mlsanity.com/paper/2508.16745/) (2025)
- [Evaluating Language Models' Evaluations of Games](https://mlsanity.com/paper/2510.10930/) (2025)
- [Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations](https://mlsanity.com/paper/2510.12699/) (2025)
- [Building Trustworthy AI by Addressing its 16+2 Desiderata with Goal-Directed Commonsense Reasoning](https://mlsanity.com/paper/2506.12667/) (2025)
- [Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference](https://mlsanity.com/paper/2601.05170/) (2026)

## Abstract (arXiv)

The reasoning abilities of large language models (LLMs) are the topic of a growing body of research in AI and cognitive science. In this paper, we probe the extent to which twenty-nine LLMs are able to distinguish logically correct inferences from logically fallacious ones. We focus on inference patterns involving conditionals (e.g., 'If Ann has a queen, then Bob has a jack') and epistemic modals (e.g., 'Ann might have an ace', 'Bob must have a king'). These inferences have been of special interest to logicians, philosophers, and linguists, since they play a central role in the fundamental human ability to reason about distal possibilities. Assessing LLMs on these inferences is thus highly relevant to the question of how much the reasoning abilities of LLMs match those of humans. All the LLMs we tested make some basic mistakes with conditionals or modals, though zero-shot chain-of-thought prompting helps them make fewer mistakes. Even the best performing LLMs make basic errors in modal reasoning, display logically inconsistent judgments across inference patterns involving epistemic modals and conditionals, and give answers about complex conditional inferences that do not match reported human judgments. These results highlight gaps in basic logical reasoning in today's LLMs.