# Benchmarking LLMs for Environmental Review and Permitting

arXiv: https://arxiv.org/abs/2407.07321 · HTML: https://mlsanity.com/paper/2407.07321/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 45 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making](https://mlsanity.com/paper/2403.16812/) (2024)
- [Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation](https://mlsanity.com/paper/2405.13622/) (2024)
- [LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking](https://mlsanity.com/paper/2308.04945/) (2023)
- [Empirical evaluation of Uncertainty Quantification in Retrieval-Augmented Language Models for Science](https://mlsanity.com/paper/2311.09358/) (2023)
- [Evalverse: Unified and Accessible Library for Large Language Model Evaluation](https://mlsanity.com/paper/2404.00943/) (2024)

## Built on by (most cited first, 4 of 4)

- [Introducing a new hyper-parameter for RAG: Context Window Utilization](https://mlsanity.com/paper/2407.19794/) (2024)
- [A Decomposition Perspective to Long-context Reasoning for LLMs](https://mlsanity.com/paper/2604.07981/) (2026)
- [Evaluating Proactive Risk Awareness of Large Language Models](https://mlsanity.com/paper/2602.20976/) (2026)
- [WuYu-EnvLE-Bench: A Benchmark for Evaluating Large Language Models in Environmental Law Enforcement](https://mlsanity.com/paper/2607.17745/) (2026)

## Abstract (arXiv)

The National Environment Policy Act (NEPA) stands as a foundational piece of environmental legislation in the United States, requiring federal agencies to consider the environmental impacts of their proposed actions. The primary mechanism for achieving this is through the preparation of Environmental Assessments (EAs) and, for significant impacts, comprehensive Environmental Impact Statements (EIS). Large Language Model (LLM)s' effectiveness in specialized domains like NEPA remains untested for adoption in federal decision-making processes. To address this gap, we present NEPA Question and Answering Dataset (NEPAQuAD), the first comprehensive benchmark derived from EIS documents, along with a modular and transparent evaluation pipeline, MAPLE, to assess LLM performance on NEPA-focused regulatory reasoning tasks. Our benchmark leverages actual EIS documents to create diverse question types, ranging from factual to complex problem-solving ones. We built a modular and transparent evaluation pipeline to test both closed- and open-source models in zero-shot or context-driven QA benchmarks. We evaluate five state-of-the-art LLMs using our framework to assess both their prior knowledge and their ability to process NEPA-specific information. The experimental results reveal that all the models consistently achieve their highest performance when provided with the gold passage as context. While comparing the other context-driven approaches for each model, Retrieval Augmented Generation (RAG)-based approaches substantially outperform PDF document contexts, indicating that neither model is well suited for long-context question-answering tasks. Our analysis suggests that NEPA-focused regulatory reasoning tasks pose a significant challenge for LLMs, particularly in terms of understanding the complex semantics and effectively processing the lengthy regulatory documents.