# LogicIF: Towards Complex Logic Instruction Following

arXiv: https://arxiv.org/abs/2508.09125 · HTML: https://mlsanity.com/paper/2508.09125/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning](https://mlsanity.com/paper/2502.01100/) (2025)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs](https://mlsanity.com/paper/2505.11423/) (2025)
- [AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios](https://mlsanity.com/paper/2505.16944/) (2025)
- [LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios](https://mlsanity.com/paper/2411.07037/) (2024)
- [Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models](https://mlsanity.com/paper/2505.14810/) (2025)
- [SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints](https://mlsanity.com/paper/2503.08669/) (2025)
- [How Many Instructions Can LLMs Follow at Once?](https://mlsanity.com/paper/2507.11538/) (2025)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [Code Simulation Challenges for Large Language Models](https://mlsanity.com/paper/2401.09074/) (2024)
- [LIFEBench: Evaluating Length Instruction Following in Large Language Models](https://mlsanity.com/paper/2505.16234/) (2025)
- [Enhancing LLM-Based Code Generation with Complexity Metrics: A Feedback-Driven Approach](https://mlsanity.com/paper/2505.23953/) (2025)
- [An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability](https://mlsanity.com/paper/2007.12520/) (2020)
- [Code Simulation as a Proxy for High-order Tasks in Large Language Models](https://mlsanity.com/paper/2502.03568/) (2025)

## Built on by (most cited first, 2 of 2)

- [LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries](https://mlsanity.com/paper/2508.15760/) (2025)
- [Sensitivity or Criterion? Measuring and Moving When Language Models Act](https://mlsanity.com/paper/2610.15415/) (2026)

## Abstract (arXiv)

Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-intensive instructions remains under-explored. We propose Logic Instruction Following (LogicIF), a task requiring models to precisely follow and simulate every logic step within complex instructions. To generate these instructions, we construct LogicIFGen, a scalable, automated framework for generating verifiable instructions from code functions, which can naturally express rich logic such as conditions, loops, nesting, and function calls. Using this framework, we curate a collection of complex code functions to construct LogicIFEval, a benchmark comprising 426 verifiable, human-examined, logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the instructions, revealing significant deficiencies in their capacity to handle instructions that involve complex logic structures. To further demonstrate the efficacy of our framework, we collect an extensive set of code functions across a wide range of logic difficulties to generate LogicIFTrain, a fully synthesized training set featuring 27,324 verifiable instructions. Our results show that a model trained on LogicIFTrain using reinforcement learning significantly outperforms the base model on both in-domain and out-of-domain benchmarks. All the data, codes and models are released at https://github.com/mianzhang/LogicIF.