# Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents

arXiv: https://arxiv.org/abs/2510.07920 · HTML: https://mlsanity.com/paper/2510.07920/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 14 of 14)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023)
- [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024)
- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model](https://mlsanity.com/paper/2402.03755/) (2024)
- [HedgeAgents: A Balanced-aware Multi-agent Financial Trading System](https://mlsanity.com/paper/2502.13165/) (2025)
- [Counterfactual Evaluation for Explainable AI](https://mlsanity.com/paper/2109.01962/) (2021)
- [FinReport: Explainable Stock Earnings Forecasting via News Factor Analyzing Model](https://mlsanity.com/paper/2403.02647/) (2024)
- [Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs](https://mlsanity.com/paper/2407.04173/) (2024)

## Built on by (most cited first, 14 of 14)

- [LiveTradeBench: Seeking Real-World Alpha with Large Language Models](https://mlsanity.com/paper/2511.03628/) (2025)
- [CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents](https://mlsanity.com/paper/2606.29771/) (2026)
- [FinTradeBench: A Financial Reasoning Benchmark for LLMs](https://mlsanity.com/paper/2603.19225/) (2026)
- [The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence](https://mlsanity.com/paper/2605.16895/) (2026)
- [All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting](https://mlsanity.com/paper/2602.17234/) (2026)
- [OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents](https://mlsanity.com/paper/2606.26350/) (2026)
- [Signal or Noise in Multi-Agent LLM-based Stock Recommendations?](https://mlsanity.com/paper/2604.17327/) (2026)
- [TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting](https://mlsanity.com/paper/2605.18843/) (2026)
- [NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models](https://mlsanity.com/paper/2605.30393/) (2026)
- [Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines](https://mlsanity.com/paper/2607.04958/) (2026)
- [HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks](https://mlsanity.com/paper/2607.18867/) (2026)
- [Agentic Empirical Asset Pricing: Methodological Foundations](https://mlsanity.com/paper/2609.00731/) (2026)
- [The Price of Thought: Does Test-Time Reasoning Pay in LLM Trading?](https://mlsanity.com/paper/2609.30705/) (2026)
- [Auditing Future-Memory Use in Point-in-Time Language Agents](https://mlsanity.com/paper/2610.18602/) (2026)

## Abstract (arXiv)

LLM-based financial agents have attracted widespread excitement for their ability to trade like human experts. However, most systems exhibit a "profit mirage": dazzling back-tested returns evaporate once the model's knowledge window ends, because of the inherent information leakage in LLMs. In this paper, we systematically quantify this leakage issue across four dimensions and release FinLake-Bench, a leakage-robust evaluation benchmark. Furthermore, to mitigate this issue, we introduce FactFin, a framework that applies counterfactual perturbations to compel LLM-based agents to learn causal drivers instead of memorized outcomes. FactFin integrates four core components: Strategy Code Generator, Retrieval-Augmented Generation, Monte Carlo Tree Search, and Counterfactual Simulator. Extensive experiments show that our method surpasses all baselines in out-of-sample generalization, delivering superior risk-adjusted performance.