mlsanityData to 14 Oct 2026

FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking

arXiv 2504.16188 ↗First posted 22 Apr 2025cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce FinNLI, a benchmark dataset for Financial Natural Language Inference (FinNLI) across diverse financial texts like SEC Filings, Annual Reports, and Earnings Call transcripts. Our dataset framework ensures diverse premise-hypothesis pairs while minimizing spurious correlations. FinNLI comprises 21,304 pairs, including a high-quality test set of 3,304 instances annotated by finance experts. Evaluations show that domain shift significantly degrades general-domain NLI performance. The highest Macro F1 scores for pre-trained (PLMs) and large language models (LLMs) baselines are 74.57% and 78.62%, respectively, highlighting the dataset's difficulty. Surprisingly, instruction-tuned financial LLMs perform poorly, suggesting limited generalizability. FinNLI exposes weaknesses in current LLMs for financial reasoning, indicating room for improvement.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 51 of its 77 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

51 of 77 references.

References · 51

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  11. 2023BloombergGPT: A Large Language Model for Financecited by 939
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
38 more
  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  3. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  4. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  5. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  7. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  8. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  9. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  10. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  11. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  12. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 148
  13. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  14. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  15. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 90
  16. 2023InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuningcited by 82
  17. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  18. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 75
  19. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  20. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  21. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  22. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  23. 2022FiNER: Financial Numeric Entity Recognition for XBRL Taggingcited by 50
  24. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 49
  25. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  26. 2020Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classificationcited by 40
  27. 2023How are Prompts Different in Terms of Sensitivity?cited by 35
  28. 2020Financial Document Causality Detection Shared Task (FinCausal 2020)cited by 31
  29. 2022SciNLI: A Corpus for Natural Language Inference on Scientific Textcited by 27
  30. 2020Self-Explaining Structures Improve NLP Modelscited by 22
  31. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 20
  32. 2022BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examplescited by 15
  33. 2024A synthetic data approach for domain generalization of NLI modelscited by 15
  34. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
  35. 2024MSciNLI: A Diverse Benchmark for Scientific Natural Language Inferencecited by 8
  36. 2024Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Modelscited by 8
  37. 2024How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamicscited by 4
  38. 2023Exploring the Impact of Corpus Diversity on Financial Pretrained Language Modelscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarkingcited by 21
  2. 2026Slogans or Stance? A Label-Light Diagnostic for Entrepreneurial-Discourse Measurement on Chinese SOE Speechescited by 0
  3. 2026MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecastingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Finance Language Model Evaluation (FLaME)
  2. 2025FINCH: Financial Intelligence using Natural language for Contextualized SQL Handling
  3. 2026FinForge: Semi-Synthetic Financial Benchmark Generation
  4. 2024FinBen: A Holistic Financial Benchmark for Large Language Models
  5. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
  6. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications
  7. 2024FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Models
  8. 2023Data-Centric Financial Large Language Models