# Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

arXiv: https://arxiv.org/abs/2608.06108 · HTML: https://mlsanity.com/paper/2608.06108/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 14 of 14)

- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [Large Language Model Agent in Financial Trading: A Survey](https://mlsanity.com/paper/2408.06361/) (2024)
- [BizBench: A Quantitative Reasoning Benchmark for Business and Finance](https://mlsanity.com/paper/2311.06602/) (2023)
- [AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets](https://mlsanity.com/paper/2512.10971/) (2025)
- [MarketSenseAI 2.0: Enhancing Stock Analysis through LLM Agents](https://mlsanity.com/paper/2502.00415/) (2025)
- [XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning](https://mlsanity.com/paper/2508.15861/) (2025)
- [FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models](https://mlsanity.com/paper/2401.02982/) (2024)
- [Real-Time Deadlines Reveal Fragile Temporal Adaptation in LLM Strategic Dialogues](https://mlsanity.com/paper/2601.13206/) (2026)
- [Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspective](https://mlsanity.com/paper/2503.18313/) (2025)
- [Distilling Analysis from Generative Models for Investment Decisions](https://mlsanity.com/paper/2410.07225/) (2024)

## Abstract (arXiv)

Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries. Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss. The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky. We ask whether the community is using the wrong ruler for consequential agents. We introduce textscInvestLogicBench, a process-native benchmark containing 201,247 documented decisions from 151 real-world investors. Each episode instantiates a PrightarrowErightarrowRrightarrowDrightarrowO trace: investor textitProfile, observable market textitEvents, investment textitReasoning, executable textitDecision, and delayed textitOutcome. The release includes profile construction, point-in-time event binding, structured logic, horizons, outcomes, and post-mortems, and supports comprehension, profile-conditioned generation, and end-to-end replay. Across four leading LLMs, logical plausibility remains near 4/5 while event grounding is only 0.8--2.8/5; return and process quality also disagree. These results expose polished but weakly grounded reasoning that outcome-only evaluation hides. We further argue that PrightarrowErightarrowRrightarrowDrightarrowO should be a data-system interface, requiring versioned profiles, temporal provenance, inspectable retrieval, decision ledgers, and replayable outcomes. Finance is our stress test for a broader class of personalized, consequential agents.