# One Agent To Rule Them All: Towards Multi-agent Conversational AI

arXiv: https://arxiv.org/abs/2203.07665 · HTML: https://mlsanity.com/paper/2203.07665/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [Response Selection for Multi-Party Conversations with Dynamic Topic Tracking](https://mlsanity.com/paper/2010.07785/) (2020)
- [Evaluating Dialogue Generation Systems via Response Selection](https://mlsanity.com/paper/2004.14302/) (2020)

## Built on by (most cited first, 11 of 11)

- [REAPER: Reasoning based Retrieval Planning for Complex RAG Systems](https://mlsanity.com/paper/2407.18553/) (2024)
- [Multi-Agents are Social Groups: Investigating Social Influence of Multiple Agents in Human-Agent Interactions](https://mlsanity.com/paper/2411.04578/) (2024)
- [Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning](https://mlsanity.com/paper/2301.10915/) (2023)
- [Redwood: Using Collision Detection to Grow a Large-Scale Intent Classification Dataset](https://mlsanity.com/paper/2204.05483/) (2022)
- [One Agent Too Many: User Perspectives on Approaches to Multi-agent Conversational AI](https://mlsanity.com/paper/2401.07123/) (2024)
- [Multi-Agent Retrieval-Augmented Framework for Evidence-Based Counterspeech Against Health Misinformation](https://mlsanity.com/paper/2507.07307/) (2025)
- [A Reference Model for Collaborative Business Intelligence Virtual Assistants](https://mlsanity.com/paper/2304.10556/) (2023)
- [Label Agnostic Pre-training for Zero-shot Text Classification](https://mlsanity.com/paper/2305.16521/) (2023)
- [TESS: A Multi-intent Parser for Conversational Multi-Agent Systems with Decentralized Natural Language Understanding Models](https://mlsanity.com/paper/2312.11828/) (2023)
- [Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels](https://mlsanity.com/paper/2512.07474/) (2025)
- [Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents](https://mlsanity.com/paper/2602.07796/) (2026)

## Abstract (arXiv)

The increasing volume of commercially available conversational agents (CAs) on the market has resulted in users being burdened with learning and adopting multiple agents to accomplish their tasks. Though prior work has explored supporting a multitude of domains within the design of a single agent, the interaction experience suffers due to the large action space of desired capabilities. To address these problems, we introduce a new task BBAI: Black-Box Agent Integration, focusing on combining the capabilities of multiple black-box CAs at scale. We explore two techniques: question agent pairing and question response pairing aimed at resolving this task. Leveraging these techniques, we design One For All (OFA), a scalable system that provides a unified interface to interact with multiple CAs. Additionally, we introduce MARS: Multi-Agent Response Selection, a new encoder model for question response pairing that jointly encodes user question and agent response pairs. We demonstrate that OFA is able to automatically and accurately integrate an ensemble of commercially available CAs spanning disparate domains. Specifically, using the MARS encoder we achieve the highest accuracy on our BBAI task, outperforming strong baselines.