mlsanityData to 14 Oct 2026

FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality

arXiv 2609.04706 ↗First posted 4 Sep 2026cs.AIcited by 1

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

A merchant's payment processor, ledger, ERP and bank feed are updated by messages that get delayed, duplicated, dropped and reordered, so for minutes at a time the four hold contradictory beliefs about the same order. An agent resolving the exception must decide whether to ship goods, re-submit a capture, refund or wait, knowing some of those cannot be undone. We present FinalityBench, an executable benchmark for that decision. It keeps a hidden canonical event log and derives each system's view from a separately faulted delivery stream, so disagreement follows from specified fault semantics rather than being authored. Grading is on executed monetary effects: an episode is scored by the merchant's terminal economic position, relative to a privileged reference told when the pending capture resolves. The corpus of 321 tasks includes 45 twin pairs (90 tasks): tasks whose four system views are identical at the decision instant, whose authoritative probes both return unknown, and whose eventual correct dispositions differ. That snapshot indistinguishability is checked under every evaluation seed rather than assumed; equivalence over all interaction traces is not claimed. Over 14,445 graded episodes from nine programmatic policies, ranking by single-task accuracy and by paired loss disagree in 7 places: a ship-on-first-sign policy is second-best by accuracy at 65.7% and worst in the suite by paired loss, because it cannot tell the two members apart. A runtime gating irreversible actions on an authoritative finality probe reaches 85.4% and, unlike every polling policy, loses nothing to pass^5; its residual loss is almost entirely one archetype, which prices finality information directly. Language models reach the same exact rate as the hand-written gate on a stratified subset, lose about twice as much money, and discover the finality-gating strategy without being told it.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 13 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

10 of 13 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents20 Jun

Other references · 9

  1. 2026STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?cited by 50
  2. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 20
  3. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  4. 2026ClawArena: Benchmarking AI Agents in Evolving Information Environmentscited by 11
  5. 2026When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agentscited by 10
  6. 2026Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failurescited by 8
  7. 2026Revisable by Design: A Theory of Streaming LLM Agent Executioncited by 5
  8. 2026AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarioscited by 5
  9. 2026FinBalance: A Multi-Document Accounting Reconciliation Benchmarkcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
  2. 2026MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
  3. 2026STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle
  4. 2026Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems
  5. 2026Governance at the Boundary: How Agent Decomposition Degrades Policy Compliance
  6. 2026Herculean: An Agentic Benchmark for Financial Intelligence
  7. 2026Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines
  8. 2026FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards