mlsanityData to 14 Oct 2026

Evaluating LLMs in Finance Requires Explicit Bias Consideration

arXiv 2602.14233 ↗First posted 15 Feb 2026cs.LG cs.AI q-fin.CPcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly integrated into financial workflows, but evaluation practice has not kept up. Finance-specific biases can inflate performance, contaminate backtests, and make reported results useless for any deployment claim. We identify five recurring biases in financial LLM applications. They include look-ahead bias, survivorship bias, narrative bias, objective bias, and cost bias. These biases break financial tasks in distinct ways and they often compound to create an illusion of validity. We reviewed 164 papers from 2023 to 2025 and found that no single bias is discussed in more than 28 percent of studies. This position paper argues that bias in financial LLM systems requires explicit attention and that structural validity should be enforced before any result is used to support a deployment claim. We propose a Structural Validity Framework and an evaluation checklist with minimal requirements for bias diagnosis and future system design. The material is available at https://github.com/Eleanorkong/Awesome-Financial-LLM-Bias-Mitigation.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 15 Feb 2026.

Builds on

47 of 71 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2025Why Language Models Hallucinatecited by 220
  7. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  8. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  9. 2024The pitfalls of next-token predictioncited by 160
  10. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  11. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
  12. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
35 more
  1. 2023Generative Language Models Exhibit Social Identity Biasescited by 111
  2. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 110
  3. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  4. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  5. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 51
  6. 2025TwinMarket: A Scalable Behavioral and Social Simulation for Financial Marketscited by 48
  7. 2025R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimizationcited by 43
  8. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 43
  9. 2025AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decaycited by 42
  10. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 40
  11. 2025FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answeringcited by 29
  12. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challengingcited by 28
  13. 2024Consistency Checks for Language Model Forecasterscited by 27
  14. 2025Kronos: A Foundation Model for the Language of Financial Marketscited by 26
  15. 2024Training on the Benchmark Is Not All You Needcited by 25
  16. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  17. 2025Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?cited by 20
  18. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  19. 2025Are Generative AI Agents Effective Personalized Financial Advisors?cited by 18
  20. 2025Your AI, Not Your View: The Bias of LLMs in Investment Analysiscited by 17
  21. 2025AlphaAgents: Large Language Model based Multi-Agents for Equity Portfolio Constructionscited by 16
  22. 2025Decision-informed Neural Networks with Large Language Model Integration for Portfolio Optimizationcited by 14
  23. 2025How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Modelscited by 9
  24. 2025AIA Forecaster: Technical Reportcited by 9
  25. 2025A Fast and Effective Solution to the Problem of Look-ahead Bias in LLMscited by 8
  26. 2025Detecting Lookahead Bias in LLM Forecastscited by 8
  27. 2025LLM-Enhanced Black-Litterman Portfolio Optimizationcited by 7
  28. 2025FinRipple: Aligning Large Language Models with Financial Market for Event Ripple Effect Awarenesscited by 6
  29. 2025Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulationscited by 5
  30. 2025Are Large Language Models Good In-context Learners for Financial Sentiment Analysis?cited by 5
  31. 2025MASCA: LLM based-Multi Agents System for Credit Assessmentcited by 3
  32. 2025Fusing Narrative Semantics for Financial Volatility Forecastingcited by 3
  33. 2025Are Foundation Models Useful for Bankruptcy Prediction?cited by 2
  34. 2025THEME: Enhancing Thematic Investing with Semantic Stock Representations and Temporal Dynamicscited by 1
  35. 2025A Methodology for Assessing the Risk of Metric Failure in LLMs Within the Financial Domaincited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Modelscited by 2
  2. 2026When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysiscited by 2
  3. 2026AlphaCrafter: Harnessing Multi-Agent Workflows for Cross-Sectional Quantitative Tradingcited by 1
  4. 2026MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecastingcited by 0
  5. 2026Forecasting With LLMs: Improved Generalization Through Feature Steeringcited by 0
  6. 2026Fund2Persona: A Framework for Building and Refining Financial Advisor Personas from Fund Disclosure Datacited by 0
  7. 2026Your AI, On a Dial: Controlling Investment Bias in LLMs with a Single Neuroncited by 0
  8. 2026Poisoning Agentic Alpha: Adversarial Vulnerabilities Across Roles and Architectures in Multi-Agent Trading Systemscited by 0
  9. 2026The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysiscited by 0
  10. 2026LiveOption: Evaluating LLM Agents in Structured Option Trading with Nonlinear Payoffscited by 0
  11. 2026FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agentscited by 0