mlsanityData to 7 Oct 2026

Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable

arXiv 2607.21340 ↗First posted 23 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In capital-markets workflows the question is rarely whether a large language model can produce a fluent draft, but whether the draft is bankable: defensible in front of a counter-party or a regulator, with the documents in hand. Existing methods address parts of that gap: open-domain QA benchmarks reward surface accuracy, and finance benchmarks (FinanceBench, FinQA, ConvFinQA) advance document-grounded and numerical QA but evaluate at the question-answer layer rather than the workflow outputs practitioners defend. We introduce CM-LRS, a Capital Markets LLM Reliability Score, evaluating outputs at the workflow-output layer across seven dimensions: factual accuracy, evidence traceability, numerical consistency, workflow completeness, source discipline, decision usefulness, and reviewability/auditability. Each is scored 0-5 against a rubric anchored on signals reviewers in regulated settings use; the aggregate is tunable to the workflow. We demonstrate CM-LRS on five workflows (DCM transaction-terms extraction, precedent retrieval, issuer profile synthesis, M&A transaction-comparable reasoning, ECM transaction-terms extraction) over public SEC EDGAR filings, a public UK takeover release, and fictional synthetic supplements, scoring four models against four independent LLM judges spanning three model families. Three findings. First, the frontier closed-source models cluster within 0.22 points on four-judge averaged CM-LRS (Sonnet 4.6 = 4.31, Opus 4.7 = 4.30, GPT-5.5 = 4.09); all four judges place the open-weights baseline (Llama 3.3 70B = 3.15) last. Second, that gap concentrates on retrieval (2.23) and synthesis (2.15), not extraction (0.84). Third, Decision Usefulness shows the widest cross-model dispersion of any dimension (4.0 points on issuer profiling) and top-tier inter-judge agreement (mean r = 0.52). Plausibility is cheap. Bankability is the bar.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

22 of 39 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2023BloombergGPT: A Large Language Model for Financecited by 923
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
10 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  4. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  5. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  6. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  7. 2024INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agentcited by 62
  8. 2023MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understandingcited by 32
  9. 2026Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategiescited by 6
  10. 2025Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysiscited by 2

Built on by

Not cited yet.