# Knowing the Form, Not the Function: Automatically Auditing Answer--Authority Decoupling in Legal Benchmarks

arXiv: https://arxiv.org/abs/2608.02621 · HTML: https://mlsanity.com/paper/2608.02621/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 8 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 5 of 5)

- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice](https://mlsanity.com/paper/2601.16669/) (2026)
- [LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation](https://mlsanity.com/paper/2510.07243/) (2025)
- [Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketing](https://mlsanity.com/paper/2510.08111/) (2025)
- [TW-LegalBench: Measuring Taiwanese Legal Understanding](https://mlsanity.com/paper/2606.18699/) (2026)

## Built on by (most cited first, 1 of 1)

- [SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges](https://mlsanity.com/paper/2610.19992/) (2026)

## Abstract (arXiv)

Legal benchmarks typically score final answers even when models also state legal authority. We test whether answer correctness can serve as a proxy for authority grounding. Under ordinary reasoning prompts that did not request statutory citations, four LLMs spontaneously produced authority markers across 238 Taiwan bar-examination items. Because each item has a verified governing provision, we automatically audit answer correctness and authority grounding jointly. The two dimensions dissociate in both directions. In criminal law, 24.0--42.4% of valid responses were answer-correct but missed the gold authority, while 15.2--21.7% were answer-incorrect but cited it. A separate statutory-retrieval probe and a permissive citation-abstention intervention further show that answer and citation behavior can move separately at the output level. Because this mismatch arises without adversarial or inconsistency-inducing prompting, answer-only scoring treats naturally occurring gold-authority misses as complete benchmark successes. Because statutory authority is structurally extractable and externally verifiable, the failure can be measured automatically. A preliminary PRC civil-law extension also observes citation-unrequested authority marking, motivating a full cross-jurisdictional joint audit. We therefore propose joint answer--authority evaluation for statute-grounded legal benchmarks.