# How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

arXiv: https://arxiv.org/abs/2601.08626 · HTML: https://mlsanity.com/paper/2601.08626/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2026.

## Built on by (most cited first, 9 of 9)

- [What Should I Cite? A RAG Benchmark for Academic Citation Prediction](https://mlsanity.com/paper/2601.14949/) (2026)
- [MultiPress: A Multi-Agent Framework for Interpretable Multimodal News Classification](https://mlsanity.com/paper/2604.03586/) (2026)
- [Rubric-Guided Process Reward for Stepwise Model Routing](https://mlsanity.com/paper/2605.29310/) (2026)
- [Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data](https://mlsanity.com/paper/2606.05122/) (2026)
- [PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning](https://mlsanity.com/paper/2606.18473/) (2026)
- [Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning](https://mlsanity.com/paper/2608.29278/) (2026)
- [PhoenixNest-Video: Evidence-Grounded Multimodal Agent Framework for Automated Video Interview Assessment](https://mlsanity.com/paper/2609.02231/) (2026)
- [JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction](https://mlsanity.com/paper/2608.29616/) (2026)
- [SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigation](https://mlsanity.com/paper/2609.16601/) (2026)

## Abstract (arXiv)

Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is difficult to evaluate automatically because scrambled sentences often admit multiple valid reorderings. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent gap between meaning-oriented generation and exact structural reconstruction, suggesting that structural robustness is not an automatic byproduct of semantic competence.