mlsanityData to 7 Oct 2026

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

arXiv 2512.16969 ↗First posted 18 Dec 2025cs.AI cs.CL cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Dec 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 24.

25 papers.

Sort:

By area:

  1. 2026InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discoverycited by 24
  2. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 23
  3. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
  4. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 4
  5. 2026SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systemscited by 4
  6. 2025SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligencecited by 3
  7. 2026Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Designcited by 3
  8. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 3
  9. 2026SciDataCopilot: An Agentic Data Preparation Framework for AGI-driven Scientific Discoverycited by 2
  10. 2026HarmonyCell: Automating Single-Cell Perturbation Modeling under Semantic and Distribution Shiftscited by 2
  11. 2026FlowPIE: Test-Time Scientific Idea Evolution with Flow-Guided Literature Explorationcited by 2
  12. 2026StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstractioncited by 2
13 more
  1. 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilitiescited by 2
  2. 2026Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scalecited by 1
  3. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuningcited by 1
  4. 2026Embodied Science: Closing the Discovery Loop with Agentic Embodied AIcited by 1
  5. 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMscited by 1
  6. 2026SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verificationcited by 1
  7. 2026TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agentscited by 1
  8. 2026AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositoriescited by 0
  9. 2026ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoningcited by 0
  10. 2026SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Taskscited by 0
  11. 2026Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generationcited by 0
  12. 2026Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinationscited by 0
  13. 2026MindFlow: Mind Supernet Powered Thinking Flows for Research Idea Innovationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
  2. 2026AI scientists produce results without reasoning scientifically
  3. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
  4. 2026Beyond the Answer: Decoding the Behavior of LLMs as Scientific Reasoners
  5. 2026Unlocking LLM Creativity in Science through Analogical Reasoning
  6. 2025Idea2Plan: Exploring AI-Powered Research Planning
  7. 2026SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
  8. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models