mlsanityData to 14 Oct 2026

ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics

arXiv 2509.00496 ↗First posted 30 Aug 2025cs.CL cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Evaluating long-form responses to research queries heavily relies on expert annotators, restricting attention to areas like AI where researchers can conveniently enlist colleagues. Yet, research expertise is abundant: survey articles consolidate knowledge spread across the literature. We introduce ResearchQA, a resource for evaluating LLM systems by distilling survey articles from 75 research fields into 21K queries and 160K rubric items. Queries and rubrics are jointly derived from survey sections, where rubric items list query-specific answer evaluation criteria, i.e., citing papers, making explanations, and describing limitations. 31 Ph.D. annotators in 8 fields judge that 90% of queries reflect Ph.D. information needs and 87% of rubric items warrant emphasis of a sentence or longer. We leverage ResearchQA to evaluate 18 systems in 7.6K head-to-heads. No parametric or retrieval-augmented system we evaluate exceeds 70% on covering rubric items, and the highest-ranking system shows 75% coverage. Error analysis reveals that the highest-ranking system fully addresses less than 11% of citation rubric items, 48% of limitation items, and 49% of comparison items. We release our data to facilitate more comprehensive multi-field evaluations.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 48 of its 183 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

48 of 183 references.

References · 48

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
36 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  2. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  3. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  4. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  6. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  7. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  8. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  9. 2023Improving Text Embeddings with Large Language Modelscited by 303
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  11. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  12. 2024Generative Representational Instruction Tuningcited by 233
  13. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  14. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  15. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  16. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  17. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  18. 2021Decontextualization: Making Sentences Stand-Alonecited by 102
  19. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  20. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  21. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  22. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  23. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  24. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  25. 2025Ai2 Scholar QA: Organized Literature Synthesis with Attributioncited by 25
  26. 2024SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluationcited by 24
  27. 2025MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documentscited by 21
  28. 2023QUDEVAL: The Evaluation of Questions Under Discussion Discourse Parsingcited by 19
  29. 2023SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generationcited by 17
  30. 2024SciDQA: A Deep Reading Comprehension Dataset over Scientific Paperscited by 15
  31. 2024KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questionscited by 11
  32. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  33. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9
  34. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8
  35. 2020CoRe: An Efficient Coarse-refined Training Framework for BERTcited by 2
  36. 2020Scaling Systematic Literature Reviews with Machine Learning Pipelinescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 45.

49 papers.

Sort:

By area:

  1. 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
  2. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 26
  3. 2025Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analyticscited by 20
  4. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  5. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
  6. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  7. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 13
  8. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  9. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 11
  10. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  11. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  12. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
37 more
  1. 2025DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completenesscited by 6
  2. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 6
  3. 2026Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarkscited by 5
  4. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 5
  5. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4
  6. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  7. 2026LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answeringcited by 4
  8. 2025Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reportscited by 3
  9. 2026Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Rankingcited by 3
  10. 2026Deep Research Pretraining via Predictive Navigationcited by 3
  11. 2026PaperGym: Rubric-Centered Evolution for Research-Plan Generationcited by 3
  12. 2026Improving Attributed Long-form Question Answering with Intent Awarenesscited by 2
  13. 2026Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systemscited by 2
  14. 2026Deep Research as Rubric for Reinforcement Learningcited by 2
  15. 2026DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Spacecited by 2
  16. 2026SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learningcited by 2
  17. 2026Training Documents Reranker with Search Rubrics for Deep Research Agentcited by 2
  18. 2026RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learningcited by 2
  19. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
  20. 2026AdaTutoRank: Learning to Rerank Document Sets via Adaptive Tutoring Optimization for RAG and Deep Researchcited by 1
  21. 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillationcited by 1
  22. 2026LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answeringcited by 0
  23. 2026DanceHA: A Multi-Agent Framework for Document-Level Aspect-Based Sentiment Analysiscited by 0
  24. 2026Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signalscited by 0
  25. 2026CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflictscited by 0
  26. 2026Hybrid Open-Ended Tri-Evolution Makes Better Deep Researchercited by 0
  27. 2026Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restorationcited by 0
  28. 2026Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillationcited by 0
  29. 2026On the Role of Citations in Preference Datacited by 0
  30. 2026Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agentscited by 0
  31. 2026Rubric Rewards from Item Response Theorycited by 0
  32. 2026From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Taskscited by 0
  33. 2026Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricscited by 0
  34. 2026How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practicescited by 0
  35. 2026RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learningcited by 0
  36. 2026CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learningcited by 0
  37. 2026Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset
  2. 2026DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
  3. 2025ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research
  4. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
  5. 2026IntelliAsk: Learning to Ask High-Quality Research Questions via RLVR
  6. 2026AnalyticsGPT: An LLM Workflow for Scientometric Question Answering
  7. 2024ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research Agents
  8. 2025SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models