mlsanityData to 14 Oct 2026

Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking

arXiv 2609.39958 ↗First posted 30 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

Corporate and investment banking teams use presentations to support credit decisions and advise clients on financing and transactions. Producing these decks requires reconciling financial data, tracing sources and turning analysis into a recommendation. We retrospectively study the development of an agentic harness combining a 27B language model, financial calculations, narrative templates and validation checks. LLM judges guide engineering changes and assess the resulting decks, raising the question of whether higher scores reflect better documents or changes in grading. In shared-session text-only grading with template markers removed, five judges score the complete system 20.4 to 33.6 points out of 95 above the same model generating directly from a short prompt. Every judge scores the system higher on all seventeen development deliverables. Margins against direct Opus generation from a short prompt range from -4.7 to +0.8 points. Judges agree on broad progress across development rounds but agree less on final-deck rankings than on pooled scores. Repeated grading also shifts scores on unchanged decks, making small improvements difficult to distinguish from judge variability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 46 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

27 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance21 May

Other references · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
14 more
  1. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  2. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  3. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  4. 2018Categorizing Variants of Goodhart's Lawcited by 149
  5. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  6. 2020With Little Power Comes Great Responsibilitycited by 142
  7. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  8. 2025PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slidescited by 86
  9. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 70
  10. 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 56
  11. 2026Rethinking the Evaluation of Harness Evolution for Agentscited by 48
  12. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  13. 2026Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelinescited by 8
  14. 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Taskscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BlueFin: Benchmarking LLM Agents on Financial Spreadsheets
  2. 2026BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
  3. 2026Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing
  4. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents
  5. 2026FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables
  6. 2026FinCom: A Financial Multi-Agent Demo with Disagree-or-Commit Deliberation
  7. 2026The Harness as the Only Mutable Surface: Compliance-Bounded Self-Evolution of LLM Agents in Credit Pipelines, with a Measured Admission Gate
  8. 2026GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation