mlsanityData to 14 Oct 2026

A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion

arXiv 2610.19999 ↗First posted 14 Oct 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation or mislabeled amendment carries into the products lawyers rely on. We report an industrial pipeline converting U.S. state legislation into strict, schema-constrained XML, and hypothesize that its accuracy comes from harness engineering, the deterministic tooling built around a model, more than the model itself. Our controlled ablation crosses three harnesses (zero-shot, a single tool-using agent, and a decomposed multi-agent pipeline) with three Claude models on 153 gold bills from two jurisdictions. The harness narrows the model gap: the weakest model in the full pipeline beats the strongest model alone (0.948 versus 0.806, paired bootstrap p less than 0.001). Because the two agentic harnesses share identical tools, their gap comes from orchestration: decomposition plus the persisted state and per-agent context it brings. Across four scored dimensions, the model spread at zero-shot (0.68 to 0.81) narrows to 0.95 at the full pipeline; the residual sits almost entirely in citation normalization, where the canonical identifier is retrieved from an external authority rather than generated from the weights, so a deterministic lookup tool supplies the exactness that a larger model does not.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

13 of 20 references.

References · 13

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  6. 2019Image-based table recognition: data, model, and evaluationcited by 237
  7. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  8. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  9. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  10. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 150
  11. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 57
  12. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 34
1 more
  1. 2026ParseBench: A Document Parsing Benchmark for AI Agentscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond the Editing Canvas: Evidence Divergence in OOXML-to-LLM Ingestion
  2. 2026Schematize: An Agentic System for Generating and Refining Information-Extraction Schemas for Legal Research
  3. 2026Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations
  4. 2026Parthenon Law: A Self-Evolving Legal-Agent Framework
  5. 2026From Naive RAG to Deep Agentic Retrieval: An Evolving Context Engineering Pipeline for Regulatory Compliance
  6. 2025AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark
  7. 2026ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction
  8. 2025An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software