# Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation

arXiv: https://arxiv.org/abs/2604.09698 · HTML: https://mlsanity.com/paper/2604.09698/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 66 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [INSPIRED: Toward Sociable Recommendation Dialog Systems](https://mlsanity.com/paper/2009.14306/) (2020)
- [SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations](https://mlsanity.com/paper/2104.08667/) (2021)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [MMREC: LLM Based Multi-Modal Recommender System](https://mlsanity.com/paper/2408.04211/) (2024)
- [MG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce](https://mlsanity.com/paper/2304.12636/) (2023)
- [Elaborative Subtopic Query Reformulation for Broad and Indirect Queries in Travel Destination Recommendation](https://mlsanity.com/paper/2410.01598/) (2024)
- [MA-DPR: Manifold-aware Distance Metrics for Dense Passage Retrieval](https://mlsanity.com/paper/2509.13562/) (2025)
- [Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments](https://mlsanity.com/paper/2510.22023/) (2025)
- [Muse: A Multimodal Conversational Recommendation Dataset with Scenario-Grounded User Profiles](https://mlsanity.com/paper/2412.18416/) (2024)
- [A Simple but Effective Elaborative Query Reformulation Approach for Natural Language Recommendation](https://mlsanity.com/paper/2510.02656/) (2025)
- [Semantic XPath: Structured Agentic Memory Access for Conversational AI](https://mlsanity.com/paper/2603.01160/) (2026)
- [SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph](https://mlsanity.com/paper/2301.01949/) (2023)
- [Design Patterns for Situated Visualization in Augmented Reality](https://mlsanity.com/paper/2307.09157/) (2023)
- [VOGUE: A Multimodal Dataset for Conversational Recommendation in Fashion](https://mlsanity.com/paper/2510.21151/) (2025)
- [Labeling Out-of-View Objects in Immersive Analytics to Support Situated Visual Searching](https://mlsanity.com/paper/2112.03354/) (2021)

## Built on by (most cited first, 5 of 5)

- [Goal-Oriented Reasoning for RAG-based Memory in Conversational Agentic LLM Systems](https://mlsanity.com/paper/2605.12213/) (2026)
- [SafeGEO: Understanding Generative Engine Optimization Risks in Recommendation Agents](https://mlsanity.com/paper/2606.28356/) (2026)
- [Temporal Order Matters for Agentic Memory: Segment Trees for Long-Horizon Agents](https://mlsanity.com/paper/2606.04555/) (2026)
- [MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulations](https://mlsanity.com/paper/2608.15844/) (2026)
- [Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants](https://mlsanity.com/paper/2610.05526/) (2026)

## Abstract (arXiv)

The growing ubiquity of Extended Reality (XR) is driving Conversational Recommendation Systems (CRS) toward visually immersive experiences. We formalize this paradigm as Immersive CRS (ICRS), where recommended items are highlighted directly in the user's scene-based visual environment and augmented with in-situ labels. While item recommendation has been widely studied, the problem of how to select and evaluate which information to present as immersive labels remains an open problem. To this end, we introduce a principled categorization of information needs into explicit intent satisfaction and proactive information needs and use these to define novel evaluation metrics for item label selection. We benchmark IR-, LLM-, and VLM-based methods across three datasets and ICRS scenarios: fashion, movie recommendation, and retail shopping. Our evaluation reveals three important limitations of existing methods: (1) they fail to leverage scenario-specific information modalities (e.g., visual cues for fashion, meta-data for retail), (2) they present redundant information that is visually inferable, and (3) they poorly anticipate users' proactive information needs from explicit dialogue alone. In summary, this work provides both a novel evaluation paradigm for in-situ item labeling in ICRS and highlights key challenges for future work.