mlsanityData to 7 Oct 2026

AI Scientists Are Only as Good as Their Evidence: A Stratified Ablation of Proprietary Data and Reasoning Skills in Drug-Asset Valuation

arXiv 2606.09556 ↗First posted 8 Jun 2026cs.AIcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

AI Scientist agents are often evaluated as if capability were mainly a function of model quality, prompting, or reasoning scaffolds. We test a different hypothesis in drug-asset valuation: for knowledge-intensive scientific decisions, the limiting factor is often the evidence substrate the agent can access. We run a controlled three-arm ablation on a production valuation agent: A is a plain web-only LLM analyst, B adds public structured tools plus a 14-dimension valuation playbook, verifier, objectivity policy and red-team, and C adds the proprietary Noah AI corpus of curated pipeline, trial and deal intelligence. Across a 13-asset stratified benchmark, B improves calibration and audit discipline: tier-in-range accuracy rises from 0.80 to 0.89 and objectivity from 3.16 to 3.30. But B does not remove the factual ceiling. Under capability-superset accounting, A and B recover only 0.25 and 0.38 of the curated gold competitive record, while C recovers 0.96; on the curated long-tail subset, C reaches 0.93 vs. 0.26/0.30. Raw blind-panel decision quality is similar for A and B (7.01 vs. 6.96), so we introduce completeness-aware decision utility: informed decision-quality = decision-quality x gold-coverage. On this metric, C reaches 7.43 vs. 1.76/2.57 for A/B. Even a perfect non-proprietary-data report would be capped at 3.83 by B's coverage. The result is not that reasoning scaffolds are unimportant; they improve calibration and discipline. Rather, proprietary evidence sets the upper bound of what the AI Scientist can know and therefore decide.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 0 of its 11 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

0 of 11 references.

No references here yet.

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can AI Agents Synthesize Scientific Conclusions?
  2. 2025FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis
  3. 2026AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
  4. 2026AI scientists produce results without reasoning scientifically
  5. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agents
  6. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Models
  7. 2026DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
  8. 2026Do AI-Native Biotechs Need Departments? Benchmarking Company World Models for AI-Driven Drug Development