# Does This Moment Justify the Recommendation? Counterfactual Behavior-Grounded Evidence Retrieval for Personalized Video Recommendation

arXiv: https://arxiv.org/abs/2609.00996 · HTML: https://mlsanity.com/paper/2609.00996/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Explainable Recommendation: A Survey and New Perspectives](https://mlsanity.com/paper/1804.11192/) (2018)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [Can I Trust Your Answer? Visually Grounded Video Question Answering](https://mlsanity.com/paper/2309.01327/) (2023)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2401.02309/) (2024)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [A Content-Driven Micro-Video Recommendation Dataset at Scale](https://mlsanity.com/paper/2309.15379/) (2023)
- [Video2GIF: Automatic Generation of Animated GIFs from Video](https://mlsanity.com/paper/1605.04850/) (2016)
- [StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos](https://mlsanity.com/paper/2304.13265/) (2023)
- [VideoCon: Robust Video-Language Alignment via Contrast Captions](https://mlsanity.com/paper/2311.10111/) (2023)
- [FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding](https://mlsanity.com/paper/2412.13441/) (2024)
- [Meta-Personalizing Vision-Language Models to Find Named Instances in Video](https://mlsanity.com/paper/2306.10169/) (2023)
- [Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval](https://mlsanity.com/paper/2502.08544/) (2025)
- [Moment Quantization for Video Temporal Grounding](https://mlsanity.com/paper/2504.02286/) (2025)
- [SHINE: Saliency-aware HIerarchical NEgative Ranking for Compositional Temporal Grounding](https://mlsanity.com/paper/2407.05118/) (2024)

## Abstract (arXiv)

Personalized video recommendation predicts user preference at the video level, while temporal video grounding localizes query-relevant moments. However, strong localization does not establish whether the retrieved moment constitutes valid evidence for recommending the video to a particular user. We study counterfactual behavior-grounded evidence retrieval, which separates where personalized evidence occurs from whether such evidence exists and evaluates whether model predictions respond consistently when that evidence is replaced. We introduce CBGER-10K, containing 5,000 controlled factual--counterfactual pairs for 3,026 users, where each pair replaces only the focal behavior-supported segment while preserving the user, temporal position, and hard distractors. We further propose CBGER, a compact framework that decouples segment-level localization from video-level evidence estimation and learns both through structured counterfactual supervision. CBGER achieves 0.4432 MRR, 0.6977 Pair Accuracy, and 0.6987 Intervention Consistency across five adapted personalized-highlight and temporal-grounding baselines. Notably, compared with QD-DETR, its MRR improvement is not statistically significant, while Pair Accuracy improves by 11.03 points. These results show that accurate temporal localization does not necessarily imply reliable personalized evidence existence, motivating explicit evaluation of Whether alongside Where.