# Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies

arXiv: https://arxiv.org/abs/2309.13063 · HTML: https://mlsanity.com/paper/2309.13063/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 93 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval](https://mlsanity.com/paper/2301.01820/) (2023)
- [TREC CAsT 2019: The Conversational Assistance Track Overview](https://mlsanity.com/paper/2003.13624/) (2020)
- [Efficiently Leveraging Multi-level User Intent for Session-based Recommendation via Atten-Mixer Network](https://mlsanity.com/paper/2206.12781/) (2022)
- [One-Shot Labeling for Automatic Relevance Estimation](https://mlsanity.com/paper/2302.11266/) (2023)
- [User-Centric Conversational Recommendation with Multi-Aspect User Modeling](https://mlsanity.com/paper/2204.09263/) (2022)
- [Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community](https://mlsanity.com/paper/2307.09751/) (2023)
- [A Computational Inflection for Scientific Discovery](https://mlsanity.com/paper/2205.02007/) (2022)
- [Generic Intent Representation in Web Search](https://mlsanity.com/paper/1907.10710/) (2019)
- [Wizard of Search Engine: Access to Information Through Conversations with Search Engines](https://mlsanity.com/paper/2105.08301/) (2021)
- [ORCAS-I: Queries Annotated with Intent using Weak Supervision](https://mlsanity.com/paper/2205.00926/) (2022)
- [Exploring Qualitative Research Using LLMs](https://mlsanity.com/paper/2306.13298/) (2023)
- [Taking Search to Task](https://mlsanity.com/paper/2301.05046/) (2023)

## Built on by (most cited first, 30 of 34)

- [Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities](https://mlsanity.com/paper/2405.10825/) (2024)
- [TnT-LLM: Text Mining at Scale with Large Language Models](https://mlsanity.com/paper/2403.12173/) (2024)
- [Large Language Models for Propaganda Span Annotation](https://mlsanity.com/paper/2311.09812/) (2023)
- [Large Action Models: From Inception to Implementation](https://mlsanity.com/paper/2412.10047/) (2024)
- [A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2404.13940/) (2024)
- [Taxonomy of User Needs and Actions](https://mlsanity.com/paper/2510.06124/) (2025)
- [From Prompt Engineering to Prompt Science With Human in the Loop](https://mlsanity.com/paper/2401.04122/) (2024)
- [Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation](https://mlsanity.com/paper/2503.16789/) (2025)
- [QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums](https://mlsanity.com/paper/2405.05345/) (2024)
- [Asking Clarifying Questions for Preference Elicitation With Large Language Models](https://mlsanity.com/paper/2510.12015/) (2025)
- [Toward Super Agent System with Hybrid AI Routers](https://mlsanity.com/paper/2504.10519/) (2025)
- [ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models](https://mlsanity.com/paper/2601.15812/) (2026)
- [Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration](https://mlsanity.com/paper/2501.12332/) (2025)
- [LLMTaxo: Leveraging Large Language Models for Constructing Taxonomy of Factual Claims from Social Media](https://mlsanity.com/paper/2504.12325/) (2025)
- [Towards End-to-End Network Intent Management with Large Language Models](https://mlsanity.com/paper/2504.13589/) (2025)
- [Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration](https://mlsanity.com/paper/2509.15786/) (2025)
- [Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Context](https://mlsanity.com/paper/2512.03671/) (2025)
- [Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset](https://mlsanity.com/paper/2602.23335/) (2026)
- [Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering](https://mlsanity.com/paper/2605.12645/) (2026)
- [AI Fiction in the Wild](https://mlsanity.com/paper/2606.22748/) (2026)
- [Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering](https://mlsanity.com/paper/2412.03815/) (2024)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering](https://mlsanity.com/paper/2509.17096/) (2025)
- [Adopt neq Adapt: Longitudinal Analyses of LLM Conversations in the Wild](https://mlsanity.com/paper/2605.29018/) (2026)
- [Towards Automated Solution Recipe Generation for Industrial Asset Management with LLM](https://mlsanity.com/paper/2407.18992/) (2024)
- [Is Our Chatbot Telling Lies? Assessing Correctness of an LLM-based Dutch Support Chatbot](https://mlsanity.com/paper/2411.00034/) (2024)
- [Efficient Intent-Based Filtering for Multi-Party Conversations Using Knowledge Distillation from LLMs](https://mlsanity.com/paper/2503.17336/) (2025)
- [Modeling User-System Behavior for Training-free Building of Private Domain Conversational Agents](https://mlsanity.com/paper/2511.10283/) (2025)
- [Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts](https://mlsanity.com/paper/2511.20736/) (2025)
- [LLM Benchmark-User Need Misalignment for Climate Change](https://mlsanity.com/paper/2603.26106/) (2026)

## Abstract (arXiv)

Log data can reveal valuable information about how users interact with Web search services, what they want, and how satisfied they are. However, analyzing user intents in log data is not easy, especially for emerging forms of Web search such as AI-driven chat. To understand user intents from log data, we need a way to label them with meaningful categories that capture their diversity and dynamics. Existing methods rely on manual or machine-learned labeling, which are either expensive or inflexible for large and dynamic datasets. We propose a novel solution using large language models (LLMs), which can generate rich and relevant concepts, descriptions, and examples for user intents. However, using LLMs to generate a user intent taxonomy and apply it for log analysis can be problematic for two main reasons: (1) such a taxonomy is not externally validated; and (2) there may be an undesirable feedback loop. To address this, we propose a new methodology with human experts and assessors to verify the quality of the LLM-generated taxonomy. We also present an end-to-end pipeline that uses an LLM with human-in-the-loop to produce, refine, and apply labels for user intent analysis in log data. We demonstrate its effectiveness by uncovering new insights into user intents from search and chat logs from the Microsoft Bing commercial search engine. The proposed work's novelty stems from the method for generating purpose-driven user intent taxonomies with strong validation. This method not only helps remove methodological and practical bottlenecks from intent-focused research, but also provides a new framework for generating, validating, and applying other kinds of taxonomies in a scalable and adaptable way with reasonable human effort.