Generative Caching for Structurally Similar Prompts and Responses

Large Language Models (LLMs) are increasingly being used to plan, reason, and execute tasks across diverse scenarios. In use cases like repeatable workflows and agentic settings, prompts are often reused with minor variations while having a similar structure for recurring tasks. This opens up opportunities for caching. However, exact prompt matching fails on such structurally similar prompts, while semantic caching may produce incorrect responses by ignoring critical differences. To address this, we introduce ourmethod, a generative cache that produces variation-aware responses for structurally similar prompts. ourmethod identifies reusable response patterns across similar prompt structures and synthesizes customized outputs for new requests. We show that ourmethod achieves 83% cache hit rate, while having minimal incorrect hits on datasets without prompt repetition. In agentic workflows, it improves cache hit rate by ~20% and reduces end-to-end execution latency by ~34% compared to standard prompt matching.