Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency. Existing benchmarks focus on domain knowledge and question answering, largely overlooking intrinsic quality review for professional documents. Such reviews rely heavily on human experts, making them costly and difficult to scale. To bridge this gap, we introduce GB/T-Bench, the first benchmark for the structured review of national standard documents. Its GB/T Review Taxonomy is a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. A controllable counterexample generation mechanism combines deterministic rules and constrained LLM rewriting to process 488 documents into 7,306 traceable review error instances for evaluation. We also develop a diagnosis-oriented evaluation protocol requiring exact matches on error location, review dimension, and error type, plus document-level coverage metrics. We further propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Experiments with 14 mainstream LLMs reveal a substantial human-LLM gap: the strongest model achieves only 0.3280 CMCS versus 0.6640 for experts. GB/T-Reviewer raises the best CMCS to 0.5094, showing the value of structured skill coordination for rule-intensive document review. This work paves the way for trustworthy AI in standardization and other high-stakes document domains.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
22 of 27 references.
References · 22
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2025Qwen-Image Technical Reportcited by 1,044
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
- 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
- 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
10 more
- 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
- 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 148
- 2023TaskWeaver: A Code-First Agent Frameworkcited by 86
- 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 61
- 2020PolicyQA: A Reading Comprehension Dataset for Privacy Policiescited by 34
- 2023MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understandingcited by 33
- 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 30
- 2024CODE-ACCORD: A Corpus of building regulatory data for rule generation towards automatic compliance checkingcited by 6
- 2021SPaR.txt, a cheap Shallow Parsing approach for Regulatory textscited by 2
- 2025HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuningcited by 1
Built on by
Not cited yet.