# Native Association: Confidence-Aware Human Perception in the Wild with a Foundation VLM

arXiv: https://arxiv.org/abs/2609.33450 · HTML: https://mlsanity.com/paper/2609.33450/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Efficient Guided Generation for Large Language Models](https://mlsanity.com/paper/2307.09702/) (2023)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [PP-OCR: A Practical Ultra Lightweight OCR System](https://mlsanity.com/paper/2009.09941/) (2020)
- [Improving Person Re-identification by Attribute and Identity Learning](https://mlsanity.com/paper/1703.07220/) (2017)
- [Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification](https://mlsanity.com/paper/1702.05891/) (2017)
- [Deep Imbalanced Attribute Classification using Visual Attention Aggregation](https://mlsanity.com/paper/1807.03903/) (2018)
- [Contextual Action Recognition with R*CNN](https://mlsanity.com/paper/1505.01197/) (2015)
- [SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimap](https://mlsanity.com/paper/2404.11335/) (2024)
- [Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion](https://mlsanity.com/paper/2312.10692/) (2023)
- [Grounding Language Models for Visual Entity Recognition](https://mlsanity.com/paper/2402.18695/) (2024)
- [Distraction-Aware Feature Learning for Human Attribute Recognition via Coarse-to-Fine Attention Mechanism](https://mlsanity.com/paper/1911.11351/) (2019)
- [A General Framework for Jersey Number Recognition in Sports Video](https://mlsanity.com/paper/2405.13896/) (2024)
- [Pedestrian Attribute Recognition: A New Benchmark Dataset and A Large Language Model Augmented Framework](https://mlsanity.com/paper/2408.09720/) (2024)

## Abstract (arXiv)

Extracting who is where, on which team, wearing which number from a broadcast frame is typically done by stitching a detector, an OCR engine, and classifiers together -- and the stitching step swaps identities under occlusion. We make association native instead: a 0.77B vision-language model (Florence-2) is fine-tuned to emit all per-person attributes as one grammar-constrained sequence, with each attribute generated inside its owner's block. Output is therefore schema-valid on every frame by construction, and no post-hoc binding step exists to attach a correctly read number to the wrong player: residual misassociation is pure perception error, ≈4× rarer than zero-shot-prompted frontier APIs' (0.057 vs. 0.21-0.24). On a frozen multi-sport test set, this single pass reaches 0.95 detection F1 (APIs: 0.65-0.75). A single extra forward pass yields a per-field confidence that supports a reject option (jersey precision 0.71rightarrow0.96 at half coverage) and routes a training-free zoom-and-re-read for small players. Surprisingly, once the grammar is learned, further parameter-efficient tuning yields no measurable gain under the adaptation configurations we test; the identical recipe on WIDER-Attribute reaches 93.1 mAP given-box, yields the first detection-coupled end-to-end results under its standard test protocol (84.5 mAP), and reproduces the same tuning result. In this regime, the gains live in the structure, not in added weights.