# Quantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking

arXiv: https://arxiv.org/abs/2604.22760 · HTML: https://mlsanity.com/paper/2604.22760/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [RestGPT: Connecting Large Language Models with Real-World RESTful APIs](https://mlsanity.com/paper/2306.06624/) (2023)
- [Further Generalizations of the Jaccard Index](https://mlsanity.com/paper/2110.09619/) (2021)
- [Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models](https://mlsanity.com/paper/2105.04651/) (2021)
- [LLM-Generated Microservice Implementations from RESTful API Definitions](https://mlsanity.com/paper/2502.09766/) (2025)
- [LogiAgent: Automated Logical Testing for REST Systems with LLM-Based Multi-Agents](https://mlsanity.com/paper/2503.15079/) (2025)
- [Service Composition in the ChatGPT Era](https://mlsanity.com/paper/2305.15788/) (2023)
- [LRASGen: LLM-based RESTful API Specification Generation](https://mlsanity.com/paper/2504.16833/) (2025)
- [Let's Discover More API Relations: A Large Language Model-based AI Chain for Unsupervised API Relation Inference](https://mlsanity.com/paper/2311.01266/) (2023)
- [Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs](https://mlsanity.com/paper/2412.13233/) (2024)
- [The Treatment of Ties in Rank-Biased Overlap](https://mlsanity.com/paper/2406.07121/) (2024)

## Abstract (arXiv)

Large language models (LLMs) increasingly operate as autonomous agents that reason over external APIs to perform complex tasks. However, their reliability and agreement remain poorly characterized. We present a unified benchmarking framework to quantify inter-LLM divergence, defined as the extent to which models differ in API discovery and ranking under identical tasks. Across 15 canonical API domains and 5 major model families, we measure pairwise and group-level agreement using set-, rank-, and consensus-based metrics including Average Overlap, Jaccard similarity, Rank-Biased Overlap, Kendall's tau, Kendall's W, and Cronbach's alpha. Results show moderate overall alignment (AO about 0.50, tau about 0.45) but strong domain dependence: structured tasks (Weather, Speech-to-Text) are stable, while open-ended tasks (Sentiment Analysis) exhibit substantially higher divergence. Volatility and consensus analyses reveal that coherence clusters around data-bound domains and degrades for abstract reasoning tasks. These insights enable reliability-aware orchestration in multi-agent systems, where consensus weighting can improve coordination among heterogeneous LLMs. Beyond performance benchmarking, our results reveal systematic failure modes in multi-agent LLM coordination, where apparent agreement can mask instability in action-relevant rankings. This hidden divergence poses a pre-deployment safety risk and motivates diagnostic benchmarks for early detection.