# EconNLI: Evaluating Large Language Models on Economics Reasoning

arXiv: https://arxiv.org/abs/2407.01212 · HTML: https://mlsanity.com/paper/2407.01212/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://mlsanity.com/paper/2401.11641/) (2024)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [A Survey of Large Language Models in Finance (FinLLMs)](https://mlsanity.com/paper/2402.02315/) (2024)
- [Is GPT-4 a Good Data Analyst?](https://mlsanity.com/paper/2305.15038/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams](https://mlsanity.com/paper/2310.08678/) (2023)
- [Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives](https://mlsanity.com/paper/2403.14341/) (2024)

## Built on by (most cited first, 8 of 8)

- [The Sociolinguistic Foundations of Language Modeling](https://mlsanity.com/paper/2407.09241/) (2024)
- [Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values](https://mlsanity.com/paper/2502.00313/) (2025)
- [CRMAgent: A Multi-Agent LLM System for E-Commerce CRM Message Template Generation](https://mlsanity.com/paper/2507.08325/) (2025)
- [Bridging Language Models and Financial Analysis](https://mlsanity.com/paper/2503.22693/) (2025)
- [Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs](https://mlsanity.com/paper/2506.00577/) (2025)
- [MetaGraph: A Large-Scale Meta-Analysis of GenAI in Financial NLP (2022-2025)](https://mlsanity.com/paper/2509.09544/) (2025)
- [RELOOP: Recursive Retrieval with Multi-Hop Reasoner and Planners for Heterogeneous QA](https://mlsanity.com/paper/2510.20505/) (2025)
- [In-Context Examples Suppress Scientific Knowledge Recall in LLMs](https://mlsanity.com/paper/2604.27540/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are widely used for writing economic analysis reports or providing financial advice, but their ability to understand economic knowledge and reason about potential results of specific economic events lacks systematic evaluation. To address this gap, we propose a new dataset, natural language inference on economic events (EconNLI), to evaluate LLMs' knowledge and reasoning abilities in the economic domain. We evaluate LLMs on (1) their ability to correctly classify whether a premise event will cause a hypothesis event and (2) their ability to generate reasonable events resulting from a given premise. Our experiments reveal that LLMs are not sophisticated in economic reasoning and may generate wrong or hallucinated answers. Our study raises awareness of the limitations of using LLMs for critical decision-making involving economic reasoning and analysis. The dataset and codes are available at https://github.com/Irenehere/EconNLI.