# Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time

arXiv: https://arxiv.org/abs/1801.02668 · HTML: https://mlsanity.com/paper/1801.02668/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2018.

## Built on by (most cited first, 10 of 10)

- [Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows](https://mlsanity.com/paper/2312.11681/) (2023)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- [Software-Based Dialogue Systems: Survey, Taxonomy and Challenges](https://mlsanity.com/paper/2106.10901/) (2021)
- [A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis](https://mlsanity.com/paper/2109.08412/) (2021)
- [Sketch-based Creativity Support Tools using Deep Learning](https://mlsanity.com/paper/2111.09991/) (2021)
- [What Types of Questions Require Conversation to Answer? A Case Study of AskReddit Questions](https://mlsanity.com/paper/2303.17710/) (2023)
- [Conversational Crowdsensing: A Parallel Intelligence Powered Novel Sensing Approach](https://mlsanity.com/paper/2402.06654/) (2024)
- [Visual Intelligence through Human Interaction](https://mlsanity.com/paper/2111.06913/) (2021)
- [Causal Inference for Chatting Handoff](https://mlsanity.com/paper/2210.02862/) (2022)
- [From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants](https://mlsanity.com/paper/2602.15859/) (2026)

## Abstract (arXiv)

Crowd-powered conversational assistants have been shown to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. A promising direction is to combine the two approaches for high quality, low latency, and low cost solutions. In this paper, we introduce Evorus, a crowd-powered conversational assistant built to automate itself over time by (i) allowing new chatbots to be easily integrated to automate more scenarios, (ii) reusing prior crowd answers, and (iii) learning to automatically approve response candidates. Our 5-month-long deployment with 80 participants and 281 conversations shows that Evorus can automate itself without compromising conversation quality. Crowd-AI architectures have long been proposed as a way to reduce cost and latency for crowd-powered systems; Evorus demonstrates how automation can be introduced successfully in a deployed system. Its architecture allows future researchers to make further innovation on the underlying automated components in the context of a deployed open domain dialog system.