mlsanityData to 7 Oct 2026

SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models

arXiv 2505.15094 ↗First posted 21 May 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval, a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification, Information-absence detection, Multi-source information integration, and Context-aware inference, through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

31 of 50 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2022A Survey on In-context Learningcited by 717
  12. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
19 more
  1. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  2. 2023Augmented Language Models: a Surveycited by 398
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  4. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  5. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  7. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  8. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 217
  9. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  10. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  11. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 124
  12. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  13. 2024ChemLLM: A Chemical Large Language Modelcited by 100
  14. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  15. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  16. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  17. 2024LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Predictioncited by 20
  18. 2025Benchmarking Retrieval-Augmented Generation for Chemistrycited by 19
  19. 2025Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Promptingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoningcited by 0
  2. 2026PathAnchor: Path-Structured Evidence for Scientific Agentscited by 0