mlsanityData to 7 Oct 2026

From Weak Task Specifications to Scientific Extraction Agents: Optimizing Task Construction

arXiv 2609.34829 ↗First posted 28 Sep 2026cs.CLcited by 0

On the map: LLM agents and tool use → Web, search, and research agents also fits: Agent skill learning and evolution

Abstract (arXiv)

Most methods that optimize LLM prompts and agent workflows assume that task-specific output schemas, extraction instructions, and evaluation criteria are predefined. For scientific extraction agents, however, a short task goal may not fully determine these components, while specifying them manually is costly. We study the upstream problem of constructing the task-specific configuration from a weak specification containing only a short goal and unannotated reference documents. Rather than treating automatic construction as a fixed preprocessing step, our framework constructs a task-specific schema, extraction instructions, and base training rubrics, then keeps schema construction and extraction instructions editable during optimization. Failure-focused updates concentrate textual-gradient feedback on lower-scoring documents, while training-time evaluation criteria adapt to recurring failures. On a heterogeneous-catalysis literature corpus, automatic construction remains improvable, and optimizing both schema construction and extraction instructions performs best across all four judge-rubric settings, with ablations and blinded human evaluation supporting the proposed formulation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 22 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

16 of 22 references.

References · 16

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Large Language Models as Optimizerscited by 830
  3. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  5. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  6. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  7. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  8. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 116
  9. 2020SciREX: A Challenge Dataset for Document-Level Information Extractioncited by 111
  10. 2023Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineeringcited by 86
  11. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 47
  12. 2023Instruct and Extract: Instruction Tuning for On-Demand Information Extractioncited by 35
4 more
  1. 2025AutoPDL: Automatic Prompt Optimization for LLM Agentscited by 27
  2. 2024Large Language Models for Scientific Information Extraction: An Empirical Study for Virologycited by 12
  3. 2026Co-Evolving LLM Evaluators and Policies via DynamicRubriccited by 8
  4. 2026AgentCAT: An LLM Agent for Extracting and Analyzing Catalytic Reaction Data from Chemical Engineering Literaturecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents
  2. 2026Enhancing LLM Instruction Following: An Evaluation-Driven Multi-Agentic Workflow for Prompt Instructions Optimization
  3. 2025Towards more Contextual Agents: An extractor-Generator Optimization Framework
  4. 2023Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis
  5. 2026AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
  6. 2026Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
  7. 2026DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents
  8. 2026Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents