Where Grounding Accuracy Lives on the IoU Curve: Label-Free Inference-Time Boundary Refinement

Vision--language models can identify the correct referent while returning an imprecise bounding box. We study whether a frozen direct-answer model can use its own prediction to allocate one additional localized observation without accessing target annotations at inference. Label-free precision refinement (LFPR) routes predicted-small regions to a higher-resolution pass, re-grounds the expression inside a context crop, admits a candidate only under fixed geometric guards, and returns a fixed coordinate-wise midpoint. We report results across three evidence tiers. On 31,921 retrospective Ref-L4 expressions, LFPR raises mAcc0.5:0.95 from 72.947% to 76.013% ([email protected] 88.531%→89.725%, [email protected] 55.788%→61.142%). A frozen transfer to 30,969 RefCOCO/RefCOCO+/RefCOCOg expressions improves every dataset at [email protected], mAcc, and mean IoU (pooled mAcc +0.645, [email protected] +0.817), while [email protected] is unchanged overall: routing alone gains +1.162 points there, but crop, guards, and fusion give back -1.192, offsetting rather than showing no strict-IoU effect. A prospective, image-disjoint Flickr30K Entities evaluation improves every endpoint (mAcc +0.973, [email protected] +1.022), more strongly under a single-box variant (mAcc +2.575, [email protected] +3.689). The same operator applied to two released grounding specialists improves every endpoint ([email protected] +1.569/+6.716 for EGM-4B/8B) at roughly twice the latency, composing with specialist training rather than replacing it. A genuine unguarded control (guard removed from the same candidates) underperforms the incumbent on every metric, showing the guard is load-bearing. Together, these results show that referent selection and boundary precision are partially separable, with different components moving opposing regions of the IoU curve -- behavior a single threshold cannot reveal.