# Whose record is this? Diagnosing and authorizing record use in personalized multimodal models

arXiv: https://arxiv.org/abs/2609.04801 · HTML: https://mlsanity.com/paper/2609.04801/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models](https://mlsanity.com/paper/2410.13360/) (2024)
- [Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agents](https://mlsanity.com/paper/2605.26497/) (2026)
- [Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness](https://mlsanity.com/paper/2407.01942/) (2024)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents](https://mlsanity.com/paper/2606.18829/) (2026)
- [MAP-Graph: Provenance-Aware Shared Memory for Multi-Agent Workflows](https://mlsanity.com/paper/2608.10509/) (2026)
- [AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes](https://mlsanity.com/paper/2305.14725/) (2023)
- [SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignment](https://mlsanity.com/paper/2410.14676/) (2024)
- [When Memory Becomes Authority: Benchmarking Authority Collapse at the Memory Consolidation Boundary](https://mlsanity.com/paper/2608.01679/) (2026)
- [MultiBind: A Benchmark for Attribute Misbinding in Multi-Subject Generation](https://mlsanity.com/paper/2603.21937/) (2026)
- [Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memory](https://mlsanity.com/paper/2607.29167/) (2026)
- [OmniHalluc-L: Counterfactual Benchmarking and Modality-Perturbation Reliability Calibration for Long-Form Omni Hallucination](https://mlsanity.com/paper/2606.03614/) (2026)
- [Unveiling the Attribute Misbinding Threat in Identity-Preserving Models](https://mlsanity.com/paper/2512.15818/) (2025)
- [SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring](https://mlsanity.com/paper/2604.25855/) (2026)
- [CAGE: Certified Authorization under Typed-Return Uncertainty for Tool-Using Agents](https://mlsanity.com/paper/2607.29190/) (2026)
- [Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models](https://mlsanity.com/paper/2608.16805/) (2026)

## Abstract (arXiv)

Contextualized visual personalization can retrieve a true record yet apply it to the wrong visual subject. We formalize when a record may condition an answer as record authorization: subject presence (P), record-edge validity (E), and answer support (S) must all hold. We call violations visual memory misbinding (VMM). We construct RecordAuth-Diag, a 3,690-case matched diagnostic suite that changes one image--record edge while holding the query, question, record text, and image multiset fixed. Card removal and nonce relabeling attribute these failures to supplied records. Raw-bank failures span Qwen-, Phi-, and Gemma-family interfaces: Gemma-3-4B-IT reaches 63.69% local unauthorized use at 25.75% clean recall. CoViP remains at 26.02%, versus 22.49% for its Qwen backbone at similar clean recall. Typed pre-generation authorization reduces Qwen card exposure on RecordAuth-Diag from 43.63% to 3.06%, while positive recall changes from 86.26% to 60.90%. Full Pwedge Ewedge S validation uses 560 localized DAVIS cases: top-1 relevance and typed authorization have comparable release (28.93% and 28.39%) but 6.79% and 0.89% unsafe release, respectively. Of the 33 additional unsafe cases removed, 27 are support, 4 edge, 2 clean, and 0 boundary cases. Thus the observed increment is an Ewedge S decision dominated by support, not an edge check alone. Appearance supplies E evidence only conditional on P; authenticated subject tokens instantiate the missing presence witness as a sufficiency control. The claims concern the evaluated contracts, not natural prevalence, consent, or visual identity