# Enterprise Benchmarks for Large Language Model Evaluation

arXiv: https://arxiv.org/abs/2410.12857 · HTML: https://mlsanity.com/paper/2410.12857/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service](https://mlsanity.com/paper/1805.01217/) (2018)
- [FiNER: Financial Numeric Entity Recognition for XBRL Tagging](https://mlsanity.com/paper/2203.06482/) (2022)
- [Plain English Summarization of Contracts](https://mlsanity.com/paper/1906.00424/) (2019)
- [Applying Deep Learning to Answer Selection: A Study and An Open Task](https://mlsanity.com/paper/1508.01585/) (2015)
- [SPEC5G: A Dataset for 5G Cellular Network Protocol Analysis](https://mlsanity.com/paper/2301.09201/) (2023)
- [EleutherAI: Going Beyond "Open Science" to "Science in the Open"](https://mlsanity.com/paper/2210.06413/) (2022)
- [Automatic Mapping of Unstructured Cyber Threat Intelligence: An Experimental Study](https://mlsanity.com/paper/2208.12144/) (2022)
- [SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications](https://mlsanity.com/paper/2404.19063/) (2024)
- [KPI-EDGAR: A Novel Dataset and Accompanying Metric for Relation Extraction from Financial Documents](https://mlsanity.com/paper/2210.09163/) (2022)
- [LAiW: A Chinese Legal Large Language Models Benchmark](https://mlsanity.com/paper/2310.05620/) (2023)
- [Generating Fact Checking Summaries for Web Claims](https://mlsanity.com/paper/2010.08570/) (2020)

## Built on by (most cited first, 5 of 5)

- [Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon](https://mlsanity.com/paper/2502.07445/) (2025)
- [LLMs Outperform Experts on Challenging Biology Benchmarks](https://mlsanity.com/paper/2505.06108/) (2025)
- [CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases](https://mlsanity.com/paper/2608.27391/) (2026)
- [The Instruction Gap: LLMs get lost in Following Instruction](https://mlsanity.com/paper/2601.03269/) (2025)
- [EvalLoop: A Methodology for Evaluation-Driven Iterative Improvement of Business AI Systems](https://mlsanity.com/paper/2607.05638/) (2026)

## Abstract (arXiv)

The advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications. Therefore, LLMs need to be able to benchmark enterprise datasets for various tasks. This work presents a systematic exploration of benchmarking strategies tailored to LLM evaluation, focusing on the utilization of domain-specific datasets and consisting of a variety of NLP tasks. The proposed evaluation framework encompasses 25 publicly available datasets from diverse enterprise domains like financial services, legal, cyber security, and climate and sustainability. The diverse performance of 13 models across different enterprise tasks highlights the importance of selecting the right model based on the specific requirements of each task. Code and prompts are available on GitHub.