# The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence arXiv: https://arxiv.org/abs/2605.16895 ยท HTML: https://mlsanity.com/paper/2605.16895/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. - Builds on 25 of its 35 references that are in the map. - First posted on arXiv 16 May 2026. ## Placed in - LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/) ## Builds on (most cited first, 25 of 25) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022) - [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023) - [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024) - [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023) - [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024) - [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024) - [Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models](https://mlsanity.com/paper/2304.07619/) (2023) - [FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making](https://mlsanity.com/paper/2407.06567/) (2024) - [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018) - [Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity](https://mlsanity.com/paper/2502.08788/) (2025) - [StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?](https://mlsanity.com/paper/2510.02209/) (2025) - [Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination](https://mlsanity.com/paper/2311.15548/) (2023) - [FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading](https://mlsanity.com/paper/2502.11433/) (2025) - [When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents](https://mlsanity.com/paper/2510.11695/) (2025) - [Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents](https://mlsanity.com/paper/2510.07920/) (2025) - [QuantHarness: Price-Driven Multi-Agent LLMs for High-Frequency Trading](https://mlsanity.com/paper/2509.09995/) (2025) - [FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use](https://mlsanity.com/paper/2603.08262/) (2026) - [Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents](https://mlsanity.com/paper/2502.17967/) (2025) - [Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge](https://mlsanity.com/paper/2504.00042/) (2025) - [A Fast and Effective Solution to the Problem of Look-ahead Bias in LLMs](https://mlsanity.com/paper/2512.06607/) (2025) - [FinRL-Meta: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance](https://mlsanity.com/paper/2112.06753/) (2021) - [Algorithms with Calibrated Machine Learning Predictions](https://mlsanity.com/paper/2502.02861/) (2025) - [TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?](https://mlsanity.com/paper/2603.00285/) (2026) ## Built on by (most cited first, 3 of 3) - [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026) - [Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines](https://mlsanity.com/paper/2607.04958/) (2026) - [EVOQUANT: Self-Evolving Verifier-Guided Strategy Optimization for Robust Quantitative Trading](https://mlsanity.com/paper/2607.12455/) (2026) ## Abstract (arXiv) End-to-end LLM trading agents have moved quickly from research curiosity to a small ecosystem of named systems, including FinCon, FinMem, TradingAgents, FinAgent, QuantAgent, and FLAG-Trader. Several of these report headline Sharpe ratios that would be material if read at face value on a deployment desk, and associated benchmarks such as FinBen report trading-task Sharpe statistics in the same range. The gap between architecture research and deployment claim has been crossed too freely on both sides of the academia--industry divide. We take a position on that gap: reported alpha from end-to-end LLM trading agents should not be treated as deployment evidence. Before such returns can support claims of deployable trading capability, they must survive structural validity tests for temporal integrity, real-world frictions, counterfactual robustness, predictive calibration, numerical execution, and multi-agent disaggregation. Current public evidence cannot yet distinguish robust predictive ability from temporal contamination, unmodeled frictions, short-window Sharpe uncertainty, narrative fitting, and parametric priors. The problem is not only evaluative but structural. Language confidence is not tradable probability, narrative reasoning is not numerical execution, and model priors may become undisclosed implicit factor exposures. We contribute a minimum reporting protocol suite, P1--P6, with tiered applicability by claim strength, and a conservative modular alternative that uses LLMs as auditable information interfaces upstream of independent calibration, risk, and execution modules. Code and reproduction harness: urlhttps://github.com/hj1650782738/Trading.