# CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

arXiv: https://arxiv.org/abs/2608.27391 · HTML: https://mlsanity.com/paper/2608.27391/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 30 of 36)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments](https://mlsanity.com/paper/2411.02305/) (2024)
- [WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting](https://mlsanity.com/paper/2405.00823/) (2024)
- [KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases](https://mlsanity.com/paper/2308.11761/) (2023)
- [Leveraging Knowledge Bases in LSTMs for Improving Machine Reading](https://mlsanity.com/paper/1902.09091/) (2019)
- [Plain English Summarization of Contracts](https://mlsanity.com/paper/1906.00424/) (2019)
- [Benchmarking Deep Search over Heterogeneous Enterprise Data](https://mlsanity.com/paper/2506.23139/) (2025)
- [Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments](https://mlsanity.com/paper/2510.27287/) (2025)
- [PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation](https://mlsanity.com/paper/2502.20377/) (2025)
- [SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications](https://mlsanity.com/paper/2404.19063/) (2024)
- [Current and Future Use of Large Language Models for Knowledge Work](https://mlsanity.com/paper/2503.16774/) (2025)
- [KPI-EDGAR: A Novel Dataset and Accompanying Metric for Relation Extraction from Financial Documents](https://mlsanity.com/paper/2210.09163/) (2022)
- [EnronQA: Towards Personalized RAG over Private Documents](https://mlsanity.com/paper/2505.00263/) (2025)
- [Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels](https://mlsanity.com/paper/2505.14925/) (2025)

## Built on by (most cited first, 1 of 1)

- [InvestigationWorlds: An Agentic Environment for Legal Investigation](https://mlsanity.com/paper/2610.04129/) (2026)

## Abstract (arXiv)

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.