When Do External Priors Help Low-Resource Structured Prediction? A Ranking-Level Alignment Study

Cold-start credit scoring (scarce labels, immature features, or minimal capacity) motivates injecting external priors (teacher models, LLMs, or domain experts) into training. We study Ranking Prior Alignment (RPA), a model-agnostic objective that aligns a model's ranking distribution to an external prior via a temperature-scaled KL loss with an annealed weight, L = Ltask + γ(t),KL(Pagent | Pmodel), and requires no external model at inference. Across tree-ensemble and multiple-instance-learning (MIL) instantiations on the public Amex dataset and an industrial merchant-credit dataset (1.5M+ merchants), we ask when such priors genuinely transfer information rather than merely act as a regularizer. Our central finding: the prior's quality π (its rank correlation with the label) governs transfer only when the alignment granularity matches the granularity at which π is measured. On cross-sample tree alignment (Amex), a high-π teacher yields +0.028 AUC over the no-alignment baseline (paired bootstrap 95% CI [0.022, 0.035]) whereas zero-information priors (uniform, random) yield none, and DeltaAUC tracks π (XGBoost Pearson r = 0.985); this replicates on a second public dataset (LendingClub). On within-bag MIL-attention alignment, the same high-π teacher transfers nothing: it is significantly worse than its own within-bag shuffle (net -0.017, 95% CI < 0, sign p = 0.021 over 10 seeds), while zero-information priors give the only significant gains: the KL term collapses to pure entropy regularization. We distill these results into a two-factor N · C decision rule (gains concentrate at low sample size and capacity and vanish as either grows) and a scale-controlled paired protocol that separates information transfer from regularization.