# Modeling shopper interest broadness with entropy-driven dialogue policy in the context of arbitrarily large product catalogs

arXiv: https://arxiv.org/abs/2509.06185 · HTML: https://mlsanity.com/paper/2509.06185/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 7 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset](https://mlsanity.com/paper/2011.06623/) (2020)
- [Embedding-based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2106.09297/) (2021)
- [Learning to Ask Appropriate Questions in Conversational Recommendation](https://mlsanity.com/paper/2105.04774/) (2021)
- [A Large Language Model Enhanced Conversational Recommender System](https://mlsanity.com/paper/2308.06212/) (2023)
- [AliMe Assist: An Intelligent Assistant for Creating an Innovative E-commerce Experience](https://mlsanity.com/paper/1801.05032/) (2018)
- [Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue](https://mlsanity.com/paper/2310.14626/) (2023)
- [Enhancing Relevance of Embedding-based Retrieval at Walmart](https://mlsanity.com/paper/2408.04884/) (2024)
- [Relevance Filtering for Embedding-based Retrieval](https://mlsanity.com/paper/2408.04887/) (2024)

## Abstract (arXiv)

Conversational recommender systems promise rich interactions for e-commerce, but balancing exploration (clarifying user needs) and exploitation (making recommendations) remains challenging, especially when deploying large language models (LLMs) with vast product catalogs. We address this challenge by modeling the breadth of user interest via the entropy of retrieval score distributions. Our method uses a neural retriever to fetch relevant items for a user query and computes the entropy of the re-ranked scores to dynamically route the dialogue policy: low-entropy (specific) queries trigger direct recommendations, whereas high-entropy (ambiguous) queries prompt exploratory questions. This simple yet effective strategy allows an LLM-driven agent to remain aware of an arbitrarily large catalog in real-time without bloating its context window.