mlsanityData to 7 Oct 2026

Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL

arXiv 2604.09470 ↗First posted 10 Apr 2026cs.CLcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Translating natural language into Jira Query Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based benchmark exists for mapping natural language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic retrieval tool that resolves natural-language mentions of categorical values through embedding-based similarity search. Among 9 frontier LLMs evaluated, single-pass models average only 43.4% execution accuracy on short natural-language queries, highlighting that text-to-JQL remains an open challenge. The agentic approach improves 7 of 9 models, with a 9.0% relative gain on the most linguistically challenging variant; in a controlled ablation isolating JiraAnchor, categorical-value accuracy rises from 48.7% to 71.7%, with component-field accuracy jumping from 16.9% to 66.2%. Our analysis identifies inherent semantic ambiguities, such as issue-type disambiguation and text-field selection, as the dominant failure modes rather than value-resolution errors, pointing to concrete directions for future work. We publicly release the benchmark, all agent transcripts, and evaluation code to support reproducibility.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 29 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

18 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems22 Nov
  2. 2025Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models22 Nov
  3. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL18 Dec

Other references · 15

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  2. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  4. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  5. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  6. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  7. 2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 164
  8. 2025ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agentscited by 36
  9. 2024Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Basescited by 23
  10. 2025MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversationscited by 21
  11. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  12. 2025Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systemscited by 15
3 more
  1. 2026Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editingcited by 3
  2. 2025Jackal: A Real-World Execution-Based Benchmark Evaluating Large Language Models on Text-to-JQL Taskscited by 2
  3. 2026DIVER: A Robust Text-to-SQL System with Dynamic Interactive Value Linking and Evidence Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL
  2. 2025Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
  3. 2025ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilities
  4. 2024NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language
  5. 2025SQL-o1: A Self-Reward Heuristic Dynamic Search Method for Text-to-SQL
  6. 2026A Semantic-Layer-Mediated Agent for Natural Language to SQL over Heterogeneous Enterprise Databases
  7. 2025LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models
  8. 2026Arming Data Agents with Tribal Knowledge