# PromptPort: A Reliability Layer for Cross-Model Structured Extraction

arXiv: https://arxiv.org/abs/2601.06151 · HTML: https://mlsanity.com/paper/2601.06151/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 3 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 1 of 1)

- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)

## Built on by (most cited first, 3 of 3)

- [PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines](https://mlsanity.com/paper/2605.18812/) (2026)
- [UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing](https://mlsanity.com/paper/2605.18796/) (2026)
- [Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science](https://mlsanity.com/paper/2606.12426/) (2026)

## Abstract (arXiv)

Structured extraction with LLMs fails in production not because models lack understanding, but because output formatting is unreliable across models and prompts. A prompt that returns clean JSON on GPT-4 may produce fenced, prose-wrapped, or malformed output on Llama, causing strict parsers to reject otherwise correct extractions. We formalize this as format collapse and introduce a dual-metric evaluation framework: ROS (strict parsing, measuring operational reliability) and CSS (post-canonicalization, measuring semantic capability). On a 37,346-example camera metadata benchmark across six model families, we find severe format collapse (for example, Gemma-2B: ROS 0.116 versus CSS 0.246) and large cross-model portability gaps (0.4 to 0.6 F1). We then present PromptPort, a reliability layer combining deterministic canonicalization with a lightweight verifier (DistilBERT) and a safe-override policy. PromptPort recovers format failures (plus 6 to 8 F1), adds verifier-driven semantic selection (plus 14 to 16 F1 beyond canonicalization), and approaches per-field oracle performance (0.890 versus 0.896 in zero-shot) without modifying base models. The method generalizes to held-out model families and provides explicit abstention when uncertain, enabling reliable structured extraction in production deployments.