# Querying Databases with Function Calling arXiv: https://arxiv.org/abs/2502.00032 ยท HTML: https://mlsanity.com/paper/2502.00032/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. - Builds on 28 of its 44 references that are in the map. - First posted on arXiv 23 Jan 2025. ## Placed in - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 28 of 28) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024) - [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021) - [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023) - [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023) - [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017) - [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022) - [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024) - [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024) - [Efficient Guided Generation for Large Language Models](https://mlsanity.com/paper/2307.09702/) (2023) - [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024) - [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024) - [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024) - [AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning](https://mlsanity.com/paper/2406.11200/) (2024) - [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024) - [Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment](https://mlsanity.com/paper/2408.06266/) (2024) - [Semantic Operators: A Declarative Model for Rich, AI-based Data Processing](https://mlsanity.com/paper/2407.11418/) (2024) - [Text2SQL is Not Enough: Unifying AI and Databases with TAG](https://mlsanity.com/paper/2408.14717/) (2024) - [DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines](https://mlsanity.com/paper/2312.13382/) (2023) - [GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications](https://mlsanity.com/paper/2404.06921/) (2024) - [StructuredRAG: JSON Response Formatting with Large Language Models](https://mlsanity.com/paper/2408.11061/) (2024) - [Agentic Information Retrieval](https://mlsanity.com/paper/2410.09713/) (2024) - [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024) - [Specifications: The missing link to making the development of LLM systems an engineering discipline](https://mlsanity.com/paper/2412.05299/) (2024) - [SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models](https://mlsanity.com/paper/2311.09818/) (2023) - [Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design](https://mlsanity.com/paper/2407.16831/) (2024) ## Built on by (most cited first, 2 of 2) - [IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering](https://mlsanity.com/paper/2602.17687/) (2026) - [OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systems](https://mlsanity.com/paper/2607.28629/) (2026) ## Abstract (arXiv) The capabilities of Large Language Models (LLMs) are rapidly accelerating largely thanks to their integration with external tools. Querying databases is among the most effective of these integrations, enabling LLMs to access private or continually updating data. While Function Calling is the most common method for interfacing external tools to LLMs, its application to database querying as a tool has been underexplored. We propose a tool definition for database querying that unifies accessing data with search queries, filters, or a combination both, as well as transforming results with aggregation and groupby operators. To evaluate its effectiveness, we conduct a study with 8 LLMs spanning 5 model families. We present a novel pipeline adapting the Gorilla LLM framework to create synthetic database schemas and queries. We primarily evaluate the models with the Exact Match of predicted and ground truth query APIs. Among the models tested, Claude 3.5 Sonnet achieves the highest performance with an Exact Match score of 74.3%, followed by GPT-4o mini at 73.7%, and GPT-4o at 71.8%. We further breakdown these results per API component utilized and across synthetic use cases. We find that LLMs are highly effective at utilizing operators on boolean properties, but struggle with text property filters. Across use cases we find robust results with the higher performing models such as GPT-4o, but significant performance variance across use cases from lower performing models. We additionally conduct ablation studies exploring the impact of parallel tool calling, adding a rationale as an argument of the tool call, using a separate tool per database collection, and tool calling with structured outputs. Our findings demonstrate the effectiveness of enabling LLMs to query databases with Function Calling. We have open-sourced our experimental code and results at github.com/weaviate/gorilla.