mlsanityData to 14 Oct 2026

Can Open-Weight Models Compete on Financial Text Comprehension?

arXiv 2608.08634 ↗First posted 9 Aug 2026cs.AI cs.CL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months. Yet their reliability on real-world financial tasks remains largely untested. We updated the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports. We also apply a new set of models on the benchmark, expanding coverage from eleven to twenty models across ten providers, including recent open-weight models such as GLM 4.7, GLM 5, Kimi K2.6, and DeepSeek V3.2, as well as Alibaba's proprietary flagship Qwen3-Max. Anthropic's Claude Opus 4.6 achieves the highest accuracy (88.4%), while Google's Gemini 2.5 Pro maintains the lowest hallucination rate (0.08%). Notably, the open-weight Kimi K2.6 ranks third in accuracy, and the non-reasoning models GLM 5 and Mistral 3 rank fourth and fifth, challenging the assumption that reasoning architectures or proprietary weights are a prerequisite for strong financial comprehension. Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. We also document a new finding: geopolitical content filters in Chinese models refuse legitimate financial questions (0.08% of attempts), sometimes without clear reason, and the refusal behavior depends on the access route as much as on the model. The complete dataset and evaluation framework are publicly available.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 43 references that are in the map. First posted on arXiv 9 Aug 2026.

Builds on

17 of 43 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2023BloombergGPT: A Large Language Model for Financecited by 939
  6. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  7. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  8. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  9. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  10. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  11. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  12. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 53
5 more
  1. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 34
  2. 2024Financial Statement Analysis with Large Language Modelscited by 31
  3. 2026ERNIE 5.0 Technical Reportcited by 11
  4. 2024AI in Investment Analysis: LLMs for Equity Stock Ratingscited by 10
  5. 2024The Structure of Financial Equity Research Reports -- Identification of the Most Frequently Asked Questions in Financial Analyst Reports to Automate Equity Research Using Llama 3 and GPT-4cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Financial Intelligence in Large Language Models: Benchmarking SuperInvesting AI with LLM Engines
  2. 2023FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models
  3. 2026Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
  4. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs
  5. 2025FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning
  6. 2026Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset
  7. 2025CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model
  8. 2026BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation