# CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

arXiv: https://arxiv.org/abs/2606.29771 · HTML: https://mlsanity.com/paper/2606.29771/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 13 of 13)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent](https://mlsanity.com/paper/2412.18174/) (2024)
- [AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets](https://mlsanity.com/paper/2512.10971/) (2025)
- [Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents](https://mlsanity.com/paper/2510.07920/) (2025)
- [LiveTradeBench: Seeking Real-World Alpha with Large Language Models](https://mlsanity.com/paper/2511.03628/) (2025)
- [Will LLMs be Professional at Fund Investment? DeepFund: A Live Arena Perspective](https://mlsanity.com/paper/2503.18313/) (2025)
- [PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management](https://mlsanity.com/paper/2605.27887/) (2026)

## Built on by (most cited first, 6 of 6)

- [ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents](https://mlsanity.com/paper/2608.30685/) (2026)
- [Agentic Empirical Asset Pricing: Methodological Foundations](https://mlsanity.com/paper/2609.00731/) (2026)
- [Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors](https://mlsanity.com/paper/2609.27051/) (2026)
- [From Search to Research: Exploring Search Scaling in Autonomous Quantitative Factor Mining](https://mlsanity.com/paper/2609.35559/) (2026)
- [LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets](https://mlsanity.com/paper/2610.09872/) (2026)
- [JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making](https://mlsanity.com/paper/2610.19589/) (2026)

## Abstract (arXiv)

LLM agents are increasingly cast as autonomous portfolio managers, yet the dominant evaluation idiom, a leaderboard of returns over a fixed window, certifies neither the soundness of an agent's process nor the durability of its edge: one period's return is dominated by the market path, and apparent alpha can dissolve once look-ahead bias and trading costs are controlled. We introduce CLQT, a closed-loop benchmark that reframes LLM trading evaluation as diagnosis rather than ranking. CLQT enforces point-in-time data access through a hard TimeGate, models institutional transaction and financing costs, scores strategy consistency across rounds, and seals every gather-analyze-decide-execute-reflect cycle into a recompute-verifiable audit chain; the same model runs as a constrained investment committee or a single autonomous orchestrator, making scaffolding an experimental variable. From the audit trail CLQT computes a five-axis capability scorecard (Coherence, Acuity, Composure, Discipline, Reliability), with coherence scored partly by a held-out LLM judge to curb self-preference bias. We validate CLQT on a contamination-controlled, year-long multi-model backtest campaign with a 13-configuration ablation grid and a four-week live broker paper-trading track on post-cutoff data. The diagnosis-first read shows that the capability leader is not the Sharpe leader; that agents' allocations systematically fail to follow their own stated analysis, a stating-versus-doing gap stable across both tracks (+0.30 backtest, +0.23 live); and that module value registers on the capability and behavioral axes when returns alone cannot separate it. Net of realistic costs, agents clear defensive baselines but do not cleanly beat the index. Credible evaluation of LLM investment agents must therefore diagnose the process rather than rank a period's return, the standard CLQT operationalizes and makes auditable.