mlsanityData to 7 Oct 2026

CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning

arXiv 2503.13517 ↗First posted 14 Mar 2025cs.CL cs.AIcited by 28

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Evaluation code and data are in https://github.com/google/curie

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 70 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

29 of 70 references.

References · 29

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
17 more
  1. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  4. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  5. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  6. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  7. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  8. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  9. 2023Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineeringcited by 86
  10. 2022Structured information extraction from complex scientific text with fine-tuned large language modelscited by 56
  11. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  12. 2023ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesiscited by 50
  13. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  14. 2024FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Modelscited by 29
  15. 2024On the Evaluation of Machine-Generated Reportscited by 23
  16. 2024Quantum Many-Body Physics Calculations with Large Language Modelscited by 17
  17. 2023M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 7.

28 papers.

Sort:

By area:

  1. 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 29
  2. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 25
  3. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 16
  4. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  5. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
  6. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  7. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  8. 2025PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problemscited by 7
  9. 2025FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMscited by 7
  10. 2025Efficient numeracy in language models through single-token number embeddingscited by 6
  11. 2026Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproductioncited by 6
  12. 2025Don't Think Twice! Over-Reasoning Impairs Confidence Calibrationcited by 5
16 more
  1. 2025CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamicscited by 5
  2. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Modelscited by 4
  3. 2025CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researcherscited by 4
  4. 2026SimMOF: AI agent for Automated MOF Simulationscited by 4
  5. 2025CROP: Circuit Retrieval and Optimization with Parameter Guidance using LLMscited by 3
  6. 2025CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Researchcited by 3
  7. 2025AI Benchmark Democratization and Carpentrycited by 3
  8. 2025Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?cited by 2
  9. 2025ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflowscited by 2
  10. 2025An MLCommons Scientific Benchmarks Ontologycited by 2
  11. 2026Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluationcited by 2
  12. 2025CrunchLLM: Multitask LLMs for Structured Business Reasoning and Outcome Predictioncited by 1
  13. 2026Toward Evaluation Frameworks for Multi-Agent Scientific AI Systemscited by 1
  14. 2026PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Paperscited by 1
  15. 2026Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articlescited by 0
  16. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
  2. 2026SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
  3. 2026SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
  4. 2025SCI-Verifier: Scientific Verifier with Thinking
  5. 2026WildSci: Advancing Scientific Reasoning from In-the-Wild Literature
  6. 2025ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning
  7. 2025OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery
  8. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning