# Semantic Feature Analysis: Improving Agents Without Searching Over Rollouts

arXiv: https://arxiv.org/abs/2604.10513 · HTML: https://mlsanity.com/paper/2604.10513/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 12 Apr 2026.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation](https://mlsanity.com/paper/2403.06988/) (2024)
- [Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directions](https://mlsanity.com/paper/2510.25445/) (2025)
- [AI-Augmented Business Process Management Systems: A Research Manifesto](https://mlsanity.com/paper/2201.12855/) (2022)
- [Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems](https://mlsanity.com/paper/2507.11277/) (2025)
- [Agentic AI Process Observability: Discovering Behavioral Variability](https://mlsanity.com/paper/2505.20127/) (2025)
- [A Taxonomy of Prompt Defects in LLM Systems](https://mlsanity.com/paper/2509.14404/) (2025)
- [Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation](https://mlsanity.com/paper/2505.07270/) (2025)

## Abstract (arXiv)

Ambiguity is an inherent property of natural-language agent specifications. When a system prompt leaves behaviour underdetermined, identical inputs follow divergent execution paths and produce inconsistent outcomes. The standard remedy is prompt optimisation: propose candidate prompts, run the agent to score them, and keep the best. This loop pays for the agent twice: once to generate candidates and again to rank them. On a tool-using agent whose rollouts cost dollars and minutes, the ranking cost dominates and budget-constrained optimisers routinely fail to find improvements. We present Semantic Feature Analysis (SFA), a pipeline that repairs agent specifications without running any search. SFA reads execution traces the agent has already produced, clusters the outputs of each workflow node, decomposes them into semantic feature classes using an extended subject-verb-object schema, ranks those features by their contribution to outcome separation using a decision tree, and injects the surviving features as corrective statements into the affected node's system prompt. Because it never ranks candidate prompts, it never spends a rollout on selection. We evaluate SFA against five prompt optimisers (GEPA, MIPROv2, SIMBA, BootstrapFewShot with random search, InferRules) and a single-reflection control, budget-matched in dollars at three budget levels across four benchmarks (IF-Bench, HotpotQA, HoVer, and GAIA). SFA consistently improves over the unmodified agent across benchmarks and budget levels, with the largest gains where rollouts are most expensive. On GAIA, where budget-constrained optimisers cannot afford to score even one candidate, SFA improves accuracy while other arms return their seed unchanged.