# Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

arXiv: https://arxiv.org/abs/2601.15322 · HTML: https://mlsanity.com/paper/2601.15322/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 17 Jan 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [Designing Heterogeneous LLM Agents for Financial Sentiment Analysis](https://mlsanity.com/paper/2401.05799/) (2024)
- [Evaluating Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2512.15567/) (2025)
- [Large Language Models: An Applied Econometric Framework](https://mlsanity.com/paper/2412.07031/) (2024)
- [Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks](https://mlsanity.com/paper/2503.16974/) (2025)
- [LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows](https://mlsanity.com/paper/2511.07585/) (2025)
- [FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos](https://mlsanity.com/paper/2512.10927/) (2025)
- [Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations](https://mlsanity.com/paper/2512.05156/) (2025)

## Built on by (most cited first, 7 of 7)

- [ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore](https://mlsanity.com/paper/2603.20625/) (2026)
- [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026)
- [FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agents](https://mlsanity.com/paper/2606.31522/) (2026)
- [DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making](https://mlsanity.com/paper/2607.20491/) (2026)
- [From Accuracy to Auditability: A Survey of Determinism in Financial AI Systems](https://mlsanity.com/paper/2605.23955/) (2026)
- [ReguSim: Evaluating LLM Agent Rule Grounding in Financial Compliance](https://mlsanity.com/paper/2608.19974/) (2026)
- [Testing and Evaluation of Agentic AI Systems In Military Command and Control](https://mlsanity.com/paper/2608.20597/) (2026)

## Abstract (arXiv)

Tool-using agents can repeat a final decision while changing their recorded execution. We introduce the Determinism-Faithfulness Assurance Harness (DFAH), a framework that distinguishes decision repeatability, trajectory agreement, and evidence-conditioned faithfulness. Task correctness requires separately qualified labels and evaluation; evidence-conditioned faithfulness was not evaluated in the historical v2 agentic experiments. The original v2 study reported 4,705 agentic runs in three synthetic financial tasks and a decision-determinism/task-label-match correlation of r = -0.11 across 21 model-benchmark configuration summaries. This statistic is reproducible from the historical configuration table, but includes a subsequently excluded portfolio fixture. It is retained as a historical description, not evidence of statistical independence, predictive uselessness, or an architectural determinism-accuracy tradeoff. Recorded decision concentration and tool-path variation do not identify hidden model strategy. This correction qualifies the historical evidence and removes the deployment recommendations derived from those unsupported interpretations. A separate corrected study, DFAH-Bench (arXiv:2607.20491), provides qualified evidence of decision/path disagreement. The contribution retained here is a measurement framework: repeatability, observable execution, evidence alignment, and correctness require distinct evidence, with explicit capture and study boundaries.