mlsanityData to 7 Oct 2026

ConvoCache: Smart Re-Use of Chatbot Responses

arXiv 2406.18133 ↗First posted 26 Jun 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present ConvoCache, a conversational caching system that solves the problem of slow and expensive generative AI models in spoken chatbots. ConvoCache finds a semantically similar prompt in the past and reuses the response. In this paper we evaluate ConvoCache on the DailyDialog dataset. We find that ConvoCache can apply a UniEval coherence threshold of 90% and respond to 89% of prompts using the cache with an average latency of 214ms, replacing LLM and voice synthesis that can take over 1s. To further reduce latency we test prefetching and find limited usefulness. Prefetching with 80% of a request leads to a 63% hit rate, and a drop in overall coherence. ConvoCache can be used with any chatbot to reduce costs by reducing usage of generative AI by up to 89%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

16 of 22 references.

References · 16

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2017Billion-scale similarity search with GPUscited by 2,477
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  9. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  10. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  11. 2023AnglE-optimized Text Embeddingscited by 98
  12. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
4 more
  1. 2019Improving Neural Conversational Models with Entropy-Based Data Filteringcited by 30
  2. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  3. 2023Personalized Predictive ASR for Latency Reduction in Voice Assistantscited by 6
  4. 2020Conversational Word Embedding for Retrieval-Based Dialog Systemcited by 2

Built on by

Not cited yet.