# From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets arXiv: https://arxiv.org/abs/2605.28359 · HTML: https://mlsanity.com/paper/2605.28359/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. - Builds on 16 of its 31 references that are in the map. - First posted on arXiv 27 May 2026. ## Placed in - LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/) ## Builds on (most cited first, 16 of 16) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021) - [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021) - [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023) - [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023) - [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022) - [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023) - [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024) - [Qlib: An AI-oriented Quantitative Investment Platform](https://mlsanity.com/paper/2009.11189/) (2020) - [Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset](https://mlsanity.com/paper/2405.10542/) (2024) - [Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking](https://mlsanity.com/paper/2505.11065/) (2025) - [OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking](https://mlsanity.com/paper/2605.03762/) (2026) ## Built on by (most cited first, 11 of 11) - [Evidence-Unit Fairness and the Limits of Query-Adaptive Sparse-Dense Fusion in Financial Document Retrieval](https://mlsanity.com/paper/2608.00183/) (2026) - [HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks](https://mlsanity.com/paper/2607.18867/) (2026) - [One Analyst Is Not Ground Truth: Grading Agent-Built Financial Models Against Observed Professional Practice](https://mlsanity.com/paper/2607.24889/) (2026) - [FORESIGHT-9: Prospective and Process-Aware Evaluation of Adaptive Trading Agents](https://mlsanity.com/paper/2608.29372/) (2026) - [What LLM Trading Agents Actually Do in Production: A Six-Month, Population-Scale Record from Two Fleets](https://mlsanity.com/paper/2609.05663/) (2026) - [The Price of Thought: Does Test-Time Reasoning Pay in LLM Trading?](https://mlsanity.com/paper/2609.30705/) (2026) - [Can Language Models Learn to Forecast Stock Prices](https://mlsanity.com/paper/2609.36914/) (2026) - [DualCast: A Dual-Path Language Model for Bimodal Financial Time-Series Forecasting](https://mlsanity.com/paper/2609.38197/) (2026) - [FactorBench: A Portfolio-Aware Benchmark for Automated Factor Mining](https://mlsanity.com/paper/2610.06947/) (2026) - [LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets](https://mlsanity.com/paper/2610.09872/) (2026) - [Auditing Future-Memory Use in Point-in-Time Language Agents](https://mlsanity.com/paper/2610.18602/) (2026) ## Abstract (arXiv) Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.