The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence
On the map: LLM agents and tool use → Financial trading and forecasting agents
Abstract (arXiv)
End-to-end LLM trading agents have moved quickly from research curiosity to a small ecosystem of named systems, including FinCon, FinMem, TradingAgents, FinAgent, QuantAgent, and FLAG-Trader. Several of these report headline Sharpe ratios that would be material if read at face value on a deployment desk, and associated benchmarks such as FinBen report trading-task Sharpe statistics in the same range. The gap between architecture research and deployment claim has been crossed too freely on both sides of the academia--industry divide. We take a position on that gap: reported alpha from end-to-end LLM trading agents should not be treated as deployment evidence. Before such returns can support claims of deployable trading capability, they must survive structural validity tests for temporal integrity, real-world frictions, counterfactual robustness, predictive calibration, numerical execution, and multi-agent disaggregation. Current public evidence cannot yet distinguish robust predictive ability from temporal contamination, unmodeled frictions, short-window Sharpe uncertainty, narrative fitting, and parametric priors. The problem is not only evaluative but structural. Language confidence is not tradable probability, narrative reasoning is not numerical execution, and model priors may become undisclosed implicit factor exposures. We contribute a minimum reporting protocol suite, P1--P6, with tiered applicability by claim strength, and a conservative modular alternative that uses LLMs as auditable information interfaces upstream of independent calibration, risk, and execution modules. Code and reproduction harness: urlhttps://github.com/hj1650782738/Trading.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 35 references that are in the map. First posted on arXiv 16 May 2026.
Builds on
25 of 35 references.
Closest ancestors · 10
Same direction, newest first
- 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use9 Mar
- 2026TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?27 Feb
- 2025When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents13 Oct
- 2025Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents9 Oct
- 2025StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?2 Oct
- 2025Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents25 Feb
- 2025FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading17 Feb
- 2024TradingAgents: Multi-Agents LLM Financial Trading Framework28 Dec
- 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making9 Jul
- 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design23 Nov
Other references · 15
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 141
- 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
- 2023Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Modelscited by 129
- 2018Troubling Trends in Machine Learning Scholarshipcited by 108
- 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneitycited by 44
- 2023Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucinationcited by 41
- 2025QuantHarness: Price-Driven Multi-Agent LLMs for High-Frequency Tradingcited by 10
- 2025Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledgecited by 8
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Agentic Trading: When LLM Agents Meet Financial Marketscited by 7
- 2026Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelinescited by 1
- 2026EVOQUANT: Self-Evolving Verifier-Guided Strategy Optimization for Robust Quantitative Tradingcited by 0
- 2026EVOQUANT: Self-Evolving Verifier-Guided Strategy Optimization for Robust Quantitative Trading14 Jul 2026
- 2026Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines6 Jul 2026
- 2026Agentic Trading: When LLM Agents Meet Financial Markets19 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems
- 2025TradeTrap: Are LLM-based Trading Agents Truly Reliable and Faithful?
- 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk
- 2026Agentic Empirical Asset Pricing: Methodological Foundations
- 2026LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets
- 2026Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness
- 2026Can Agentic Trading Systems Pay for Their Own Intelligence?
- 2026Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents