# From Weak Task Specifications to Scientific Extraction Agents: Optimizing Task Construction

arXiv: https://arxiv.org/abs/2609.34829 · HTML: https://mlsanity.com/paper/2609.34829/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 16 of 16)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [Offline Training of Language Model Agents with Functions as Learnable Weights](https://mlsanity.com/paper/2402.11359/) (2024)
- [Instruct and Extract: Instruction Tuning for On-Demand Information Extraction](https://mlsanity.com/paper/2310.16040/) (2023)
- [AutoPDL: Automatic Prompt Optimization for LLM Agents](https://mlsanity.com/paper/2504.04365/) (2025)
- [Large Language Models for Scientific Information Extraction: An Empirical Study for Virology](https://mlsanity.com/paper/2401.10040/) (2024)
- [Co-Evolving LLM Evaluators and Policies via DynamicRubric](https://mlsanity.com/paper/2607.20083/) (2026)
- [AgentCAT: An LLM Agent for Extracting and Analyzing Catalytic Reaction Data from Chemical Engineering Literature](https://mlsanity.com/paper/2602.18479/) (2026)

## Abstract (arXiv)

Most methods that optimize LLM prompts and agent workflows assume that task-specific output schemas, extraction instructions, and evaluation criteria are predefined. For scientific extraction agents, however, a short task goal may not fully determine these components, while specifying them manually is costly. We study the upstream problem of constructing the task-specific configuration from a weak specification containing only a short goal and unannotated reference documents. Rather than treating automatic construction as a fixed preprocessing step, our framework constructs a task-specific schema, extraction instructions, and base training rubrics, then keeps schema construction and extraction instructions editable during optimization. Failure-focused updates concentrate textual-gradient feedback on lower-scoring documents, while training-time evaluation criteria adapt to recurring failures. On a heterogeneous-catalysis literature corpus, automatic construction remains improvable, and optimizing both schema construction and extraction instructions performs best across all four judge-rubric settings, with ablations and blinded human evaluation supporting the proposed formulation.