mlsanityData to 14 Oct 2026

Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage

arXiv 2605.30826 ↗First posted 29 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Biomedical NER is deceptively simple for modern LLMs: plausible biomedical mentions are easy to surface, but corpus-convention correctness depends on annotation conventions, span boundaries, entity granularity, and type schemas. Multi-LLM agreement is a salience signal, not corpus-convention correctness. We introduce a candidate-level panel-output benchmark for panel-surfaced candidate verification, where the unit is an aligned candidate surfaced by an explicitly defined multi-model panel rather than a standalone extractor output. The benchmark aligns eight LLMs' predictions over five public biomedical NER datasets into a candidate master table. BioConCal is an in-domain supervised scorer that instantiates this layer with inference-time gold-free agreement, mention, surface-availability, and document features for a fixed candidate stream. In domain, BioConCal improves AUROC from 0.753 for raw agreement to 0.910. At a validation-selected 0.95 precision target it selects 1,340 candidates at empirical test precision 0.939, compared with 293 for raw agreement. This corresponds to candidate-level recall 0.592 and corpus-level recall 0.523 against a within-panel row-label ceiling of 0.883. The main benefit is not recovering entities missed by every panel member, but reshaping a noisy panel stream into a higher-yield review queue. Under entity-type shift, thresholds require target-domain validation, and exact character localization remains a separate deterministic post-processing step.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 39 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

14 of 39 references.

References · 14

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  7. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  8. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  9. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  10. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  11. 2022BERN2: an advanced neural biomedical named entity recognition and normalization toolcited by 42
  12. 2024LLMs are not Zero-Shot Reasoners for Biomedical Information Extractioncited by 9
2 more
  1. 2025A Case Study of Scalable Content Annotation Using Multi-LLM Consensus and Human Reviewcited by 7
  2. 2025Do LLMs Surpass Encoders for Biomedical NER?cited by 6

Built on by

Not cited yet.