RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluation
On the map: LLM reasoning → Reasoning for agents, planning and code
Abstract (arXiv)
Multi-step symbolic reasoning is essential for robust financial analysis, yet most benchmarks neglect intermediate reasoning steps. FinChain introduced verifiable Chain-of-Thought (CoT) evaluation but is limited to English. FINESSE-Bench includes a Russian block but relies on multiple-choice questions without step-level supervision. We present RusFinChain, the first Russian-language symbolic benchmark for verifiable CoT reasoning in finance. It spans 17 domains, 172 topics, and comprises 5,280 parameterized examples from executable Python templates, ensuring contamination-free evaluation. Each example includes a gold-standard reasoning chain with intermediate numeric values for automatic verification. We also introduce enhanced metrics: Fuzzy Numeric Alignment and Soft-Attention Alignment. We evaluate 8 open-weight LLMs on a stratified sample, generating 8,100 responses. Results reveal a substantial reasoning gap: models achieve Hard F1 of ~0.65 for step alignment, but only ~29% of final answers are correct. Our fuzzy and soft metrics show stronger correlation with final-answer correctness (Spearman rho approx 0.48) than the original ChainEval (rho approx 0.38-0.46), demonstrating superior diagnostic power. We release dataset, code, and evaluation framework to foster verifiable financial AI for the Russian-speaking community.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 34 of its 43 references that are in the map. First posted on arXiv 1 Jul 2026.
Builds on
34 of 43 references.
Closest ancestors · 5
Same direction, newest first
- 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging6 Jun
- 2025FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning3 Jun
- 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains16 Nov
- 2023BizBench: A Quantitative Reasoning Benchmark for Business and Finance11 Nov
- 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering7 Oct
Other references · 29
- 2017Attention Is All You Needcited by 59,932
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023Mistral 7Bcited by 3,065
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2023BloombergGPT: A Large Language Model for Financecited by 923
- 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
- 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
17 more
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
- 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
- 2023Faithful Chain-of-Thought Reasoningcited by 285
- 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
- 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
- 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
- 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
- 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
- 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 72
- 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 65
- 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
- 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
- 2023DeltaScore: Fine-Grained Story Evaluation with Perturbationscited by 9
- 2025Finance Language Model Evaluation (FLaME)cited by 4
- 2026FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Modelscited by 4
- 2026The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systemscited by 2
- 2026FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agentscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles
- 2026V-FiLLM: Verified Financial LLM Reasoning Benchmark
- 2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance
- 2024FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning
- 2026Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
- 2025FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning
- 2026Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Text
- 2026FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems