# Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition

arXiv: https://arxiv.org/abs/2602.14955 · HTML: https://mlsanity.com/paper/2602.14955/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 16 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)](https://mlsanity.com/paper/2401.02500/) (2024)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks](https://mlsanity.com/paper/2403.11085/) (2024)
- [InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation](https://mlsanity.com/paper/2407.06423/) (2024)
- [LiP-LLM: Integrating Linear Programming and dependency graph with Large Language Models for multi-robot task planning](https://mlsanity.com/paper/2410.21040/) (2024)
- [FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data](https://mlsanity.com/paper/2509.02473/) (2025)
- [Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study](https://mlsanity.com/paper/2507.23589/) (2025)
- [Haste Makes Waste: Evaluating Planning Abilities of LLMs for Efficient and Feasible Multitasking with Time Constraints Between Actions](https://mlsanity.com/paper/2503.02238/) (2025)
- [Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation](https://mlsanity.com/paper/2507.02253/) (2025)
- [Prompt2DAG: A Modular Methodology for LLM-Based Data Enrichment Pipeline Generation](https://mlsanity.com/paper/2509.13487/) (2025)

## Abstract (arXiv)

We present a domain-grounded framework and benchmark for tool-aware plan generation in contact centers, where answering a query for business insights, our target use case, requires decomposing it into executable steps over structured tools (Text2SQL (T2S)/Snowflake) and unstructured tools (RAG/transcripts) with explicit depends_on for parallelism. Our contributions are threefold: (i) a reference-based plan evaluation framework operating in two modes - a metric-wise evaluator spanning seven dimensions (e.g., tool-prompt alignment, query adherence) and a one-shot evaluator; (ii) a data curation methodology that iteratively refines plans via an evaluator->optimizer loop to produce high-quality plan lineages (ordered plan revisions) while reducing manual effort; and (iii) a large-scale study of 14 LLMs across sizes and families for their ability to decompose queries into step-by-step, executable, and tool-assigned plans, evaluated under prompts with and without lineage. Empirically, LLMs struggle on compound queries and on plans exceeding 4 steps (typically 5-15); the best total metric score reaches 84.8% (Claude-3-7-Sonnet), while the strongest one-shot match rate at the "A+" tier (Extremely Good, Very Good) is only 49.75% (o3-mini). Plan lineage yields mixed gains overall but benefits several top models and improves step executability for many. Our results highlight persistent gaps in tool-understanding, especially in tool-prompt alignment and tool-usage completeness, and show that shorter, simpler plans are markedly easier. The framework and findings provide a reproducible path for assessing and improving agentic planning with tools for answering data-analysis queries in contact-center settings.