mlsanityData to 14 Oct 2026

Evaluating LLMs' Reasoning Over Ordered Procedural Steps

arXiv 2511.04688 ↗First posted 25 Oct 2025cs.CL cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Reasoning over procedural sequences, where the order of steps directly impacts outcomes, is a critical capability for large language models (LLMs). In this work, we study the task of reconstructing globally ordered sequences from shuffled procedural steps, using a curated dataset of food recipes, a domain where correct sequencing is essential for task success. We evaluate several LLMs under zero-shot and few-shot settings and present a comprehensive evaluation framework that adapts established metrics from ranking and sequence alignment. These include Kendall's Tau, Normalized Longest Common Subsequence (NLCS), and Normalized Edit Distance (NED), which capture complementary aspects of ordering quality. Our analysis shows that model performance declines with increasing sequence length, reflecting the added complexity of longer procedures. We also find that greater step displacement in the input, corresponding to more severe shuffling, leads to further degradation. These findings highlight the limitations of current LLMs in procedural reasoning, especially with longer and more disordered inputs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 19 references that are in the map. First posted on arXiv 25 Oct 2025.

Builds on

14 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure4 Oct
  3. 2023STEPS: A Benchmark for Order Reasoning in Sequential Tasks7 Jun

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023Mistral 7Bcited by 3,164
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  6. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  7. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  8. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  9. 2019Generating Personalized Recipes from Historical User Preferencescited by 53
  10. 2024EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoningcited by 22
  11. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability
  2. 2025seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs
  3. 2026EvalQReason: A Framework for Step-Level Reasoning Evaluation in Large Language Models
  4. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  5. 2024CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans
  6. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
  7. 2024Premise Order Matters in Reasoning with Large Language Models
  8. 2024MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks