# GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

arXiv: https://arxiv.org/abs/2603.29112 · HTML: https://mlsanity.com/paper/2603.29112/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models](https://mlsanity.com/paper/2402.04614/) (2024)
- [Personalization of Large Language Models: A Survey](https://mlsanity.com/paper/2411.00027/) (2024)
- [KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems](https://mlsanity.com/paper/2202.10842/) (2022)
- [Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2305.13112/) (2023)
- [Transparent and Scrutable Recommendations Using Natural Language User Profiles](https://mlsanity.com/paper/2402.05810/) (2024)
- [iAgent: LLM Agent as a Shield between User and Recommender Systems](https://mlsanity.com/paper/2502.14662/) (2025)
- [Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMs](https://mlsanity.com/paper/2503.09382/) (2025)
- [Large Language Models as Evaluators for Recommendation Explanations](https://mlsanity.com/paper/2406.03248/) (2024)
- [End-to-end Training for Recommendation with Language-based User Profiles](https://mlsanity.com/paper/2410.18870/) (2024)
- [Intrinsic User-Centric Interpretability through Global Mixture of Experts](https://mlsanity.com/paper/2402.02933/) (2024)
- [ALPBench: A Benchmark for Attribution-level Long-term Personal Behavior Understanding](https://mlsanity.com/paper/2602.03056/) (2026)
- [RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models](https://mlsanity.com/paper/2412.11068/) (2024)

## Built on by (most cited first, 3 of 3)

- [SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context](https://mlsanity.com/paper/2606.26654/) (2026)
- [PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks](https://mlsanity.com/paper/2608.21381/) (2026)
- [KuaFu: Compressing Long User Behavior into Understanding at Billion Scale](https://mlsanity.com/paper/2609.31045/) (2026)

## Abstract (arXiv)

We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in recommendation systems. Unlike traditional RecSys benchmarks that focus on item prediction accuracy, our benchmark evaluates how well LLMs can extract and verify user interests from engagement data. We propose two novel metric families: Interest Groundedness (IG), decomposed into precision and recall components to separately penalize hallucinated interest categories and reward coverage, and Interest Specificity (IS), which assesses the distinctiveness of verified LLM-predicted user profiles. We release a synthetic dataset constructed on real user interactions on a global short-form video platform. Our dataset contains both implicit and explicit engagement signals and rich textual descriptions. We validate our dataset fidelity against user surveys, and evaluate eight open-weight LLMs spanning 7B to 235B parameters, together with three proprietary frontier models (GPT-5, Claude 4.6, and Gemini 3.5 Flash). Our findings reveal performance bottlenecks in current LLMs, particularly their limited ability to accurately count and attribute engagement signals across heterogeneous interaction types.