# Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge

arXiv: https://arxiv.org/abs/2610.08675 · HTML: https://mlsanity.com/paper/2610.08675/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 7 of 7)

- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [JEV-as-a-Judge: Accept When Confident, Escalate When Unsure](https://mlsanity.com/paper/2609.26550/) (2026)
- [Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It](https://mlsanity.com/paper/2609.26758/) (2026)
- [Improved LLM Agents for Financial Document Question Answering](https://mlsanity.com/paper/2506.08726/) (2025)
- [VeriFin: A Neurosymbolic Framework for Verifying LLM-Generated Financial Claims](https://mlsanity.com/paper/2608.10213/) (2026)
- [FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings](https://mlsanity.com/paper/2608.07400/) (2026)
- [BudgetVerify: Budget-Tiered Verification for Financial QA](https://mlsanity.com/paper/2609.33052/) (2026)

## Abstract (arXiv)

Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role. We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. A signed-number-at-pointer baseline explains most recovery over exact quotation checks. To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable.