# Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments

arXiv: https://arxiv.org/abs/2510.22023 · HTML: https://mlsanity.com/paper/2510.22023/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2025.

## Built on by (most cited first, 9 of 9)

- [Semantic XPath: Structured Agentic Memory Access for Conversational AI](https://mlsanity.com/paper/2603.01160/) (2026)
- [Bayesian Active Learning with Gaussian Processes Guided by LLM Relevance Scoring for Dense Passage Retrieval](https://mlsanity.com/paper/2604.17906/) (2026)
- [Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation](https://mlsanity.com/paper/2604.09698/) (2026)
- [Goal-Oriented Reasoning for RAG-based Memory in Conversational Agentic LLM Systems](https://mlsanity.com/paper/2605.12213/) (2026)
- [SafeGEO: Understanding Generative Engine Optimization Risks in Recommendation Agents](https://mlsanity.com/paper/2606.28356/) (2026)
- [Temporal Order Matters for Agentic Memory: Segment Trees for Long-Horizon Agents](https://mlsanity.com/paper/2606.04555/) (2026)
- [MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulations](https://mlsanity.com/paper/2608.15844/) (2026)
- [BoundaryMORPH: Budgeted Reranking via Active Set Selection for Diffuse Retrieval](https://mlsanity.com/paper/2609.27213/) (2026)
- [Have I Scene This Before? Spatially Grounded Conversational Memory for Complex Queries in Egocentric Assistants](https://mlsanity.com/paper/2610.05526/) (2026)

## Abstract (arXiv)

Natural Language Recommendation (NLRec) generates item suggestions based on the relevance between user-issued NL requests and NL item description passages. Existing NLRec approaches often use Dense Retrieval (DR) to compute item relevance scores from aggregation of inner products between user request embeddings and relevant passage embeddings. However, DR views the request as the sole relevance label, thus leading to a unimodal scoring function centered on the query embedding that is often a weak proxy for query relevance. To better capture the potential multimodal distribution of the relevance scoring function that may arise from complex NLRec data, we propose GPR-LLM that uses Gaussian Process Regression (GPR) with LLM relevance judgments for a subset of candidate passages. Experiments on four NLRec datasets and two LLM backbones demonstrate that GPR-LLM with an RBF kernel, capable of modeling multimodal relevance scoring functions, consistently outperforms simpler unimodal kernels (dot product, cosine similarity), as well as baseline methods including DR, cross-encoder, and pointwise LLM-based relevance scoring by up to 65%. Overall, GPR-LLM provides an efficient and effective approach to NLRec within a minimal LLM labeling budget.