Capital Markets LLM Reliability Score (CM-LRS): From Plausible to Bankable
Not on the map. Why?
Abstract (arXiv)
In capital-markets workflows the question is rarely whether a large language model can produce a fluent draft, but whether the draft is bankable: defensible in front of a counter-party or a regulator, with the documents in hand. Existing methods address parts of that gap: open-domain QA benchmarks reward surface accuracy, and finance benchmarks (FinanceBench, FinQA, ConvFinQA) advance document-grounded and numerical QA but evaluate at the question-answer layer rather than the workflow outputs practitioners defend. We introduce CM-LRS, a Capital Markets LLM Reliability Score, evaluating outputs at the workflow-output layer across seven dimensions: factual accuracy, evidence traceability, numerical consistency, workflow completeness, source discipline, decision usefulness, and reviewability/auditability. Each is scored 0-5 against a rubric anchored on signals reviewers in regulated settings use; the aggregate is tunable to the workflow. We demonstrate CM-LRS on five workflows (DCM transaction-terms extraction, precedent retrieval, issuer profile synthesis, M&A transaction-comparable reasoning, ECM transaction-terms extraction) over public SEC EDGAR filings, a public UK takeover release, and fictional synthetic supplements, scoring four models against four independent LLM judges spanning three model families. Three findings. First, the frontier closed-source models cluster within 0.22 points on four-judge averaged CM-LRS (Sonnet 4.6 = 4.31, Opus 4.7 = 4.30, GPT-5.5 = 4.09); all four judges place the open-weights baseline (Llama 3.3 70B = 3.15) last. Second, that gap concentrates on retrieval (2.23) and synthesis (2.15), not extraction (0.84). Third, Decision Usefulness shows the widest cross-model dispersion of any dimension (4.0 points on issuer profiling) and top-tier inter-judge agreement (mean r = 0.52). Plausibility is cheap. Bankability is the bar.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 23 Jul 2026.
Builds on
22 of 39 references.
References · 22
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2022Survey of Hallucination in Natural Language Generationcited by 2,056
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2023BloombergGPT: A Large Language Model for Financecited by 923
- 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
10 more
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
- 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
- 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
- 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
- 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
- 2024INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agentcited by 62
- 2023MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understandingcited by 32
- 2026Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategiescited by 6
- 2025Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysiscited by 2
Built on by
Not cited yet.