mlsanityData to 7 Oct 2026

RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluation

arXiv 2607.01388 ↗First posted 1 Jul 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Multi-step symbolic reasoning is essential for robust financial analysis, yet most benchmarks neglect intermediate reasoning steps. FinChain introduced verifiable Chain-of-Thought (CoT) evaluation but is limited to English. FINESSE-Bench includes a Russian block but relies on multiple-choice questions without step-level supervision. We present RusFinChain, the first Russian-language symbolic benchmark for verifiable CoT reasoning in finance. It spans 17 domains, 172 topics, and comprises 5,280 parameterized examples from executable Python templates, ensuring contamination-free evaluation. Each example includes a gold-standard reasoning chain with intermediate numeric values for automatic verification. We also introduce enhanced metrics: Fuzzy Numeric Alignment and Soft-Attention Alignment. We evaluate 8 open-weight LLMs on a stratified sample, generating 8,100 responses. Results reveal a substantial reasoning gap: models achieve Hard F1 of ~0.65 for step alignment, but only ~29% of final answers are correct. Our fuzzy and soft metrics show stronger correlation with final-answer correctness (Spearman rho approx 0.48) than the original ChainEval (rho approx 0.38-0.46), demonstrating superior diagnostic power. We release dataset, code, and evaluation framework to foster verifiable financial AI for the Russian-speaking community.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 34 of its 43 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

34 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging6 Jun
  2. 2025FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning3 Jun
  3. 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains16 Nov
  4. 2023BizBench: A Quantitative Reasoning Benchmark for Business and Finance11 Nov
  5. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering7 Oct

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Mistral 7Bcited by 3,065
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  11. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  12. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
17 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  2. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  3. 2023Faithful Chain-of-Thought Reasoningcited by 285
  4. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  5. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  6. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
  7. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  8. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  9. 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 72
  10. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 65
  11. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  12. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  13. 2023DeltaScore: Fine-Grained Story Evaluation with Perturbationscited by 9
  14. 2025Finance Language Model Evaluation (FLaME)cited by 4
  15. 2026FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Modelscited by 4
  16. 2026The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systemscited by 2
  17. 2026FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles
  2. 2026V-FiLLM: Verified Financial LLM Reasoning Benchmark
  3. 2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance
  4. 2024FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning
  5. 2026Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
  6. 2025FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning
  7. 2026Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Text
  8. 2026FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems