mlsanityData to 14 Oct 2026

BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting

arXiv 2605.17937 ↗First posted 18 May 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Quantitative backtesting is essential for evaluating trading strategies but remains hampered by high technical barriers and limited scalability. While Large Language Models (LLMs) offer a transformative path to automate this complex, interdisciplinary workflow through advanced code generation, tool usage, and agentic planning, the practical realization is significantly challenged by the current lack of a large-scale benchmark dedicated to automated quantitative backtesting, which hinders progress in this field. To bridge this critical gap, we introduce BacktestBench, the first large-scale benchmark for automated quantitative backtesting. Built from over 6 million real market records, it comprises 18,246 meticulously annotated question-answering pairs across four task categories: metrics calculation, ticker selection, strategy selection, and parameter confirmation. We also propose AutoBacktest, a robust multi-agent baseline that translates natural language strategies into reproducible backtests by coordinating a Summarizer for semantic factor extraction, a Retriever for validated SQL generation, and a Coder for Python backtesting implementation. Our evaluation on 23 mainstream LLMs, complemented by targeted ablations, identifies key factors that influence end-to-end performance and highlights the importance of grounded verification and standardized indicator representations.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 39 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

11 of 39 references.

References · 11

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  5. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  6. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  7. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
  8. 2023Generating Synergistic Formulaic Alpha Collections via Reinforcement Learningcited by 39
  9. 2024Automate Strategy Finding with LLM in Quant Investmentcited by 26
  10. 2024AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Frameworkcited by 12
  11. 2025Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamicscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Agentic Trading: When LLM Agents Meet Financial Marketscited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2026QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
  2. 2026QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models
  3. 2026Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
  4. 2024FinBen: A Holistic Financial Benchmark for Large Language Models
  5. 2026Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models
  6. 2026QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code
  7. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs
  8. 2025LiveTradeBench: Seeking Real-World Alpha with Large Language Models