mlsanityData to 14 Oct 2026

RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines

arXiv 2609.13389 ↗First posted 11 Sep 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Mapping textual specifications into formal representations is essential for ensuring the correctness of protocol designs and implementations. LLM-generated mappings, used for networking security or testing, are assumed to capture a perfect understanding of the specification, which may not hold in practice. The goal of this paper is to assess the extent to which LLMs can interpret the specification correctly. We examine the degree to which an LLM's implicit representation of a finite-state transition system-defined via natural language descriptions-aligns with a manually generated ground-truth model. We designed 4 tasks and 1482 task queries for 16 protocols. We evaluated different judge biases, observed the inherent difficulty gaps between tasks, looked into the effect of 4 context types, and the influence of protocol characteristics. Our work contributes to a step toward verifying whether LLMs can really be trusted in FSM (Finite State Machine) reasoning of protocol specifications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 7 of its 33 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

7 of 33 references.

References · 7

  1. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  2. 2024Evaluating the World Model Implicit in a Generative Modelcited by 113
  3. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 47
  4. 2022Automated Attack Synthesis by Extracting Finite State Machines from Protocol Specification Documentscited by 8
  5. 2025RFCAudit: An LLM Agent for Functional Bug Detection in Network Protocolscited by 7
  6. 2025LLM-Assisted Model-Based Fuzzing of Protocol Implementationscited by 3
  7. 2024A Formal Analysis of SCTP: Attack Synthesis and Patch Verificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unleashing the Power of LLM to Infer State Machine from the Protocol Implementation
  2. 2025An Agentic Flow for Finite State Machine Extraction using Prompt Chaining
  3. 2026LLM-FSM: Scaling Large Language Models for Finite-State Reasoning in RTL Code Generation
  4. 2025Automated Network Protocol Testing with LLM Agents
  5. 2025Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference
  6. 2025Retrieval Augmented Generation Based LLM Evaluation For Protocol State Machine Inference With Chain-of-Thought Reasoning
  7. 2026PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces
  8. 2026Fewer Assumptions by Design: A Reusable Skill for LLM-Assisted Verus Verification