# QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation

arXiv: https://arxiv.org/abs/2210.15718 · HTML: https://mlsanity.com/paper/2210.15718/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [Learning-to-Rank with BERT in TF-Ranking](https://mlsanity.com/paper/2004.08476/) (2020)
- [ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search](https://mlsanity.com/paper/2006.05324/) (2020)
- [Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling](https://mlsanity.com/paper/2207.03030/) (2022)
- [Query Understanding via Intent Description Generation](https://mlsanity.com/paper/2008.10889/) (2020)

## Built on by (most cited first, 21 of 21)

- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement](https://mlsanity.com/paper/2402.11060/) (2024)
- [INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning](https://mlsanity.com/paper/2401.06532/) (2024)
- [Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLM](https://mlsanity.com/paper/2312.15450/) (2023)
- [SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM](https://mlsanity.com/paper/2403.04735/) (2024)
- [Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration](https://mlsanity.com/paper/2405.16546/) (2024)
- [Unified Generative and Discriminative Training for Multi-modal Large Language Models](https://mlsanity.com/paper/2411.00304/) (2024)
- [Question Decomposition for Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.00355/) (2025)
- [SSFF: Investigating LLM Predictive Capabilities for Startup Success through a Multi-Agent Framework with Enhanced Explainability and Performance](https://mlsanity.com/paper/2405.19456/) (2024)
- [Context Tuning for Retrieval Augmented Generation](https://mlsanity.com/paper/2312.05708/) (2023)
- [Meta Knowledge for Retrieval Augmented Large Language Models](https://mlsanity.com/paper/2408.09017/) (2024)
- [Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning](https://mlsanity.com/paper/2411.13045/) (2024)
- [Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components](https://mlsanity.com/paper/2407.00997/) (2024)
- [Event-Centric Query Expansion in Web Search](https://mlsanity.com/paper/2305.19019/) (2023)
- [Agentic Multi-Source Grounding for Enhanced Query Intent Understanding: A DoorDash Case Study](https://mlsanity.com/paper/2603.01486/) (2026)
- [Sherlock Your Queries: Learning to Ask the Right Questions for Dialogue-Based Retrieval](https://mlsanity.com/paper/2510.18659/) (2025)
- [Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.22694/) (2025)
- [Sensitivity-Aware Retrieval-Augmented Intent Clarification](https://mlsanity.com/paper/2603.06025/) (2026)
- [A Unified Structured Query Understanding Framework for Industrial Semantic Search](https://mlsanity.com/paper/2605.27441/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown impressive results on a variety of text understanding tasks. Search queries though pose a unique challenge, given their short-length and lack of nuance or context. Complicated feature engineering efforts do not always lead to downstream improvements as their performance benefits may be offset by increased complexity of knowledge distillation. Thus, in this paper we make the following contributions: (1) We demonstrate that Retrieval Augmentation of queries provides LLMs with valuable additional context enabling improved understanding. While Retrieval Augmentation typically increases latency of LMs (thus hurting distillation efficacy), (2) we provide a practical and effective way of distilling Retrieval Augmentation LLMs. Specifically, we use a novel two-stage distillation approach that allows us to carry over the gains of retrieval augmentation, without suffering the increased compute typically associated with it. (3) We demonstrate the benefits of the proposed approach (QUILL) on a billion-scale, real-world query understanding system resulting in huge gains. Via extensive experiments, including on public benchmarks, we believe this work offers a recipe for practical use of retrieval-augmented query understanding.