# PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding

arXiv: https://arxiv.org/abs/2210.12308 · HTML: https://mlsanity.com/paper/2210.12308/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Scalable Zero-shot Entity Linking with Dense Entity Retrieval](https://mlsanity.com/paper/1911.03814/) (2019)
- [Efficient One-Pass End-to-End Entity Linking for Questions](https://mlsanity.com/paper/2010.02413/) (2020)
- [Improving Multi-turn Dialogue Modelling with Utterance ReWriter](https://mlsanity.com/paper/1906.07004/) (2019)
- [Multi-task Retrieval for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2101.00117/) (2021)
- [Pre-trained Language Model for Web-scale Retrieval in Baidu Search](https://mlsanity.com/paper/2106.03373/) (2021)
- [Learning to Attend, Copy, and Generate for Session-Based Query Suggestion](https://mlsanity.com/paper/1708.03418/) (2017)
- [Feedback-Based Self-Learning in Large-Scale Conversational AI Agents](https://mlsanity.com/paper/1911.02557/) (2019)
- [A review of on-device fully neural end-to-end automatic speech recognition algorithms](https://mlsanity.com/paper/2012.07974/) (2020)
- [CREAD: Combined Resolution of Ellipses and Anaphora in Dialogues](https://mlsanity.com/paper/2105.09914/) (2021)
- [Pre-Training for Query Rewriting in A Spoken Language Understanding System](https://mlsanity.com/paper/2002.05607/) (2020)
- [Personalized Query Rewriting in Conversational AI Agents](https://mlsanity.com/paper/2011.04748/) (2020)

## Built on by (most cited first, 3 of 3)

- [Graph Meets LLM: A Novel Approach to Collaborative Filtering for Robust Conversational Understanding](https://mlsanity.com/paper/2305.14449/) (2023)
- [KG-ECO: Knowledge Graph Enhanced Entity Correction for Query Rewriting](https://mlsanity.com/paper/2302.10454/) (2023)
- [CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models](https://mlsanity.com/paper/2403.02745/) (2024)

## Abstract (arXiv)

Conversational understanding is an integral part of modern intelligent devices. In a large fraction of the global traffic from customers using smart digital assistants, frictions in dialogues may be attributed to incorrect understanding of the entities in a customer's query due to factors including ambiguous mentions, mispronunciation, background noise and faulty on-device signal processing. Such errors are compounded by two common deficiencies from intelligent devices namely, (1) the device not being tailored to individual customers, and (2) the device responses being unaware of the context in the conversation session. Viewing this problem via the lens of retrieval-based search engines, we build and evaluate a scalable entity correction system, PENTATRON. The system leverages a parametric transformer-based language model to learn patterns from in-session customer-device interactions coupled with a non-parametric personalized entity index to compute the correct query, which aids downstream components in reasoning about the best response. In addition to establishing baselines and demonstrating the value of personalized and context-aware systems, we use multitasking to learn the domain of the correct entity. We also investigate the utility of language model prompts. Through extensive experiments, we show a significant upward movement of the key metric (Exact Match) by up to 500.97% (relative to the baseline).