mlsanityData to 14 Oct 2026

Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

arXiv 2601.15322 ↗First posted 17 Jan 2026cs.AI cs.CLcited by 7

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

Tool-using agents can repeat a final decision while changing their recorded execution. We introduce the Determinism-Faithfulness Assurance Harness (DFAH), a framework that distinguishes decision repeatability, trajectory agreement, and evidence-conditioned faithfulness. Task correctness requires separately qualified labels and evaluation; evidence-conditioned faithfulness was not evaluated in the historical v2 agentic experiments. The original v2 study reported 4,705 agentic runs in three synthetic financial tasks and a decision-determinism/task-label-match correlation of r = -0.11 across 21 model-benchmark configuration summaries. This statistic is reproducible from the historical configuration table, but includes a subsequently excluded portfolio fixture. It is retained as a historical description, not evidence of statistical independence, predictive uselessness, or an architectural determinism-accuracy tradeoff. Recorded decision concentration and tool-path variation do not identify hidden model strategy. This correction qualifies the historical evidence and removes the deployment recommendations derived from those unsupported interpretations. A separate corrected study, DFAH-Bench (arXiv:2607.20491), provides qualified evidence of decision/path disagreement. The contribution retained here is a measurement framework: repeatability, observable execution, evidence alignment, and correctness require distinct evidence, with explicit capture and study boundaries.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 38 references that are in the map. First posted on arXiv 17 Jan 2026.

Builds on

17 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysis11 Jan

Other references · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  7. 2023BloombergGPT: A Large Language Model for Financecited by 939
  8. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  9. 2023FinGPT: Open-Source Financial Large Language Modelscited by 253
  10. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  11. 2025Evaluating Large Language Models in Scientific Discoverycited by 42
  12. 2024Large Language Models: An Applied Econometric Frameworkcited by 32
4 more
  1. 2025Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Taskscited by 20
  2. 2025LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflowscited by 7
  3. 2025FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videoscited by 5
  4. 2025Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Models
  2. 2026Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing
  3. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk
  4. 2026Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
  5. 2026Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems
  6. 2026Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints
  7. 2025Benchmarking LLM Agents for Wealth-Management Workflows
  8. 2026The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications