# Evaluating LLM-Based Grant Proposal Review via Structured Perturbations

arXiv: https://arxiv.org/abs/2603.08281 · HTML: https://mlsanity.com/paper/2603.08281/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 79 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://mlsanity.com/paper/2406.16253/) (2024)
- [Less is More for Long Document Summary Evaluation by LLMs](https://mlsanity.com/paper/2309.07382/) (2023)
- [Towards Understanding the Robustness of LLM-based Evaluations under Perturbations](https://mlsanity.com/paper/2412.09269/) (2024)
- [Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages](https://mlsanity.com/paper/2601.06341/) (2026)
- [Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts](https://mlsanity.com/paper/2504.21117/) (2025)
- [Can REF output quality scores be assigned by AI? Experimental evidence](https://mlsanity.com/paper/2212.08041/) (2022)

## Built on by (most cited first, 1 of 1)

- [NVAITC AI Scientist: A Governed End-to-End Research System -- A Hypertension GWAS Case Study](https://mlsanity.com/paper/2607.11084/) (2026)

## Abstract (arXiv)

As AI-assisted grant proposals outpace manual review capacity in a kind of ``Malthusian trap'' for the research ecosystem, this paper investigates the capabilities and limitations of LLM-based grant reviewing for high-stakes evaluation. Using six EPSRC proposals, we develop a perturbation-based framework probing LLM sensitivity across six quality axes: funding, timeline, competency, alignment, clarity, and impact. We compare three review architectures: single-pass review, section-by-section analysis, and a 'Council of Personas' ensemble emulating expert panels. The section-level approach significantly outperforms alternatives in both detection rate and scoring reliability, while the computationally expensive council method performs no better than baseline. Detection varies substantially by perturbation type, with alignment issues readily identified but clarity flaws largely missed by all systems. Human evaluation shows LLM feedback is largely valid but skewed toward compliance checking over holistic assessment. We conclude that current LLMs may provide supplementary value within EPSRC review but exhibit high variability and misaligned review priorities. We release our code and any non-protected data.