# RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines

arXiv: https://arxiv.org/abs/2609.13389 · HTML: https://mlsanity.com/paper/2609.13389/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 7 of 7)

- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [Evaluating the World Model Implicit in a Generative Model](https://mlsanity.com/paper/2406.03689/) (2024)
- [Can LLM Graph Reasoning Generalize beyond Pattern Memorization?](https://mlsanity.com/paper/2406.15992/) (2024)
- [Automated Attack Synthesis by Extracting Finite State Machines from Protocol Specification Documents](https://mlsanity.com/paper/2202.09470/) (2022)
- [RFCAudit: An LLM Agent for Functional Bug Detection in Network Protocols](https://mlsanity.com/paper/2506.00714/) (2025)
- [LLM-Assisted Model-Based Fuzzing of Protocol Implementations](https://mlsanity.com/paper/2508.01750/) (2025)
- [A Formal Analysis of SCTP: Attack Synthesis and Patch Verification](https://mlsanity.com/paper/2403.05663/) (2024)

## Abstract (arXiv)

Mapping textual specifications into formal representations is essential for ensuring the correctness of protocol designs and implementations. LLM-generated mappings, used for networking security or testing, are assumed to capture a perfect understanding of the specification, which may not hold in practice. The goal of this paper is to assess the extent to which LLMs can interpret the specification correctly. We examine the degree to which an LLM's implicit representation of a finite-state transition system-defined via natural language descriptions-aligns with a manually generated ground-truth model. We designed 4 tasks and 1482 task queries for 16 protocols. We evaluated different judge biases, observed the inherent difficulty gaps between tasks, looked into the effect of 4 context types, and the influence of protocol characteristics. Our work contributes to a step toward verifying whether LLMs can really be trusted in FSM (Finite State Machine) reasoning of protocol specifications.