Reliable conformal novelty detection at the decision boundary

Novelty detection via conformal p-values and BH procedure provides distribution-free global false discovery rate (FDR) control. We present here fundamental limits of this approach by showing that it does not produce reliable detection at the decision boundary. We study boundary false discovery rate (bFDR), the probability that the least extreme reported novelty is in fact a null observation. We first show that the support line (SL) procedure, controlling the bFDR in the continuous independent framework, fails to control the bFDR in the conformal case. We then present several modifications of the SL procedure that restore reliability at the decision boundary, by controlling the bFDR, with specific improvements in situations where many novelties are expected (adaptive procedures) and when the calibration sample is too small with respect to the test sample (subsampled procedures). Numerical experiments with both synthetic and real data support our findings and show the relevance of the new proposed approach.