# YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering

arXiv: https://arxiv.org/abs/2505.14279 · HTML: https://mlsanity.com/paper/2505.14279/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 92 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 30 of 55)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)

## Built on by (most cited first, 16 of 16)

- [FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use](https://mlsanity.com/paper/2603.08262/) (2026)
- [ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?](https://mlsanity.com/paper/2511.22978/) (2025)
- [RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension](https://mlsanity.com/paper/2601.14289/) (2026)
- [OmniQEC: discovering practical quantum error-correcting codes by an AI scientist](https://mlsanity.com/paper/2607.25865/) (2026)
- [DeepResearchEco: A Recursive Agentic Workflow for Complex Scientific Question Answering in Ecology](https://mlsanity.com/paper/2507.10522/) (2025)
- [Judge Model for Large-scale Multimodality Benchmarks](https://mlsanity.com/paper/2601.06106/) (2026)
- [Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarks](https://mlsanity.com/paper/2605.03472/) (2026)
- [COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning](https://mlsanity.com/paper/2608.30109/) (2026)
- [CLINB: A Climate Intelligence Benchmark for Foundational Models](https://mlsanity.com/paper/2511.11597/) (2025)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://mlsanity.com/paper/2601.13729/) (2026)
- [Structure-BiEval: A Self-Supervised, Dual-Track Framework for Decoupling Structure and Content in LLM Evaluation for Web Information Systems](https://mlsanity.com/paper/2601.19923/) (2026)
- [LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering](https://mlsanity.com/paper/2602.23603/) (2026)
- [GRAFT: Graph-Distilled Generative Retrieval for Facet-Aware Scientific Literature Exploration](https://mlsanity.com/paper/2608.22381/) (2026)
- [Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judges](https://mlsanity.com/paper/2608.29127/) (2026)
- [ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedback](https://mlsanity.com/paper/2609.09072/) (2026)
- [Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals](https://mlsanity.com/paper/2610.16008/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) drive scientific question-answering on modern search engines, yet their evaluation robustness remains underexplored. We introduce YESciEval, an open-source framework that combines fine-grained rubric-based assessment with reinforcement learning to mitigate optimism bias in LLM evaluators. We release multidisciplinary scienceQ&A datasets, including adversarial variants, with evaluation scores from multiple LLMs. Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation. By advancing reliable LLM-as-a-judge models, this work supports AI alignment and fosters robust, transparent evaluation essential for scientific inquiry.