mlsanityData to 13 Oct 2026

ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors

arXiv 2608.01204 ↗First posted 2 Aug 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Conversational investment advisors influence not only what users know, but also how they make subsequent decisions as market conditions evolve. Existing evaluations primarily assess response quality or observed outcomes, leaving the long-horizon pathway from advisor language to investor behavior difficult to audit. We introduce ShiJianBench, an offline framework for evaluating conversational investment advisors through matched investor trajectories under fixed historical market feedback. At its core is a multi-agent investor simulator with explicit evolving state variables, motive-driven deliberation, long-term memory, and dialogue-grounded updates. The simulator is calibrated against aggregate behavioral patterns from 7,199 real users, and advisor policies are evaluated using separate investor-side, service-side, and content-side metrics under a hard compliance gate. Experiments on Chinese fund-market traces from 2021 to 2026 identify a stable leading group of LLM advisors that combines substantially stronger personalized content with competitive investor-side trajectory outcomes. These results reveal a systematic distinction between producing a high-quality response and delivering an effective long-horizon intervention, motivating trajectory-aware evaluation of conversational advisors.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 49 of its 103 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

49 of 103 references.

References · 49

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,541
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,218
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  5. 2018Self-Attentive Sequential Recommendationcited by 1,175
  6. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 958
  7. 2023BloombergGPT: A Large Language Model for Financecited by 932
  8. 2015Session-based Recommendations with Recurrent Neural Networkscited by 688
  9. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 553
  10. 2018Personalized Top-N Sequential Recommendation via Convolutional Sequence Embeddingcited by 475
  11. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 405
  12. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 357
37 more
  1. 2018Session-based Recommendation with Graph Neural Networkscited by 312
  2. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 264
  3. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 247
  4. 2023Large Language Models in Finance: A Surveycited by 226
  5. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 218
  6. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  7. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  8. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 164
  9. 2020A Survey on Conversational Recommender Systemscited by 143
  10. 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
  11. 2020TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Trackingcited by 130
  12. 2016Off-policy evaluation for slate recommendationcited by 102
  13. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 92
  14. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  15. 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
  16. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 65
  17. 2024INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agentcited by 63
  18. 2023The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challengescited by 59
  19. 2019Estimating Treatment Effects with Causal Forests: An Applicationcited by 58
  20. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 54
  21. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 47
  22. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  23. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 46
  24. 2025StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?cited by 43
  25. 2025Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profilescited by 40
  26. 2024LLM-Powered User Simulator for Recommender Systemcited by 38
  27. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  28. 2021CRSLab: An Open-Source Toolkit for Building Conversational Recommender Systemcited by 34
  29. 2025SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluationcited by 32
  30. 2019ConvLab: Multi-Domain End-to-End Dialog System Platformcited by 29
  31. 2023A Survey on User Behavior Modeling in Recommender Systemscited by 26
  32. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 26
  33. 2021RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystemscited by 21
  34. 2022Multi-Behavior Sequential Recommendation with Temporal Graph Transformercited by 20
  35. 2020Adversarial Counterfactual Learning and Evaluation for Recommender Systemcited by 16
  36. 2025User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluationcited by 16
  37. 2018Goal-oriented Dialogue Policy Learning from Failurescited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Are LLMs Good Financial User Simulators? Multi-view Investor Logic Alignment (MILA)cited by 0