GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding

Standard dual-encoder vision-language models that map images and text to deterministic points on a shared unit hypersphere through ℓ2 normalization typically expose neither aleatoric uncertainty (cross-modal ambiguity) nor epistemic uncertainty (lack of training-distribution support). Existing post-hoc methods either recover at most one of the two uncertainty components, or ignore the hyperspherical geometry of these models' embeddings. We propose GeoFlowVLM as a post-hoc adapter that learns the joint distribution of paired ℓ2-normalised dual-encoder VLM embeddings on the product hypersphere mathbbSd-1 × mathbbSd-1 via Riemannian flow matching with a single masked velocity field. A consistency result shows that, in the population limit, the trained network exposes the joint flow and both cross-modal conditional flows as valid Riemannian flow-matching velocity fields on their respective domains. We derive two quantities from this single model: a conditional retrieval entropy that quantifies aleatoric ambiguity with a decision-theoretic interpretation via a Fano-type bound, and a marginal-typicality epistemic score justified by an exact chain-rule decomposition of the joint NLL. This decomposition isolates a cross-modal pointwise-mutual-information term that is structurally discriminative rather than epistemic, and is empirically the only consistently uninformative standalone component. Empirically, the entropy tracks Recall@1 with near-ideal monotonic calibration across three retrieval benchmarks in both directions, and the marginal-typicality sum yields consistently calibrated selective accuracy across four zero-shot classification benchmarks.