mlsanityData to 14 Oct 2026

YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering

arXiv 2505.14279 ↗First posted 20 May 2025cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) drive scientific question-answering on modern search engines, yet their evaluation robustness remains underexplored. We introduce YESciEval, an open-source framework that combines fine-grained rubric-based assessment with reinforcement learning to mitigate optimism bias in LLM evaluators. We release multidisciplinary scienceQ&A datasets, including adversarial variants, with evaluation scores from multiple LLMs. Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation. By advancing reliable LLM-as-a-judge models, this work supports AI alignment and fosters robust, transparent evaluation essential for scientific inquiry.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 55 of its 92 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

55 of 92 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  13. 2018CoQA: A Conversational Question Answering Challengecited by 683
  14. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  15. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  16. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  17. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  18. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  19. 2018Generating Natural Language Adversarial Examplescited by 424
  20. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  21. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  22. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  23. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  24. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  25. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  26. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  27. 2023The False Promise of Imitating Proprietary LLMscited by 263
  28. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  29. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  30. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  31. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  32. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  33. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  34. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  35. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  36. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  37. 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
  38. 2023DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graphcited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Usecited by 10
  2. 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 9
  3. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4
  4. 2026OmniQEC: discovering practical quantum error-correcting codes by an AI scientistcited by 4
  5. 2025DeepResearchEco: A Recursive Agentic Workflow for Complex Scientific Question Answering in Ecologycited by 1
  6. 2026Judge Model for Large-scale Multimodality Benchmarkscited by 1
  7. 2026Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarkscited by 1
  8. 2026COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learningcited by 1
  9. 2025CLINB: A Climate Intelligence Benchmark for Foundational Modelscited by 0
  10. 2026On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluationcited by 0
  11. 2026Structure-BiEval: A Self-Supervised, Dual-Track Framework for Decoupling Structure and Content in LLM Evaluation for Web Information Systemscited by 0
  12. 2026LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answeringcited by 0
4 more
  1. 2026GRAFT: Graph-Distilled Generative Retrieval for Facet-Aware Scientific Literature Explorationcited by 0
  2. 2026Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judgescited by 0
  3. 2026ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedbackcited by 0
  4. 2026Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
  2. 2026SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation
  3. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth
  4. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
  5. 2025MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers
  6. 2025An Open and Reproducible Deep Research Agent for Long-Form Question Answering
  7. 2025Deliberative Searcher: Improving LLM Reliability via Reinforcement Learning with constraints
  8. 2025Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems