mlsanityData to 7 Oct 2026

Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoning

arXiv 2608.22839 ↗First posted 24 Aug 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) are increasingly used for scientific decision support, yet reliable confidence estimation remains difficult in black-box settings. We study uncertainty estimation for hierarchical taxonomic reasoning generated by a black-box LLM in a long-tailed biodiversity monitoring pipeline. Using proxy features extracted by an open-source tool LLM, we train lightweight supervised estimators with hierarchy-aware supervision to predict rank-wise correctness. Across three tool LLMs, the supervised estimators consistently outperform a token-likelihood baseline for micro discrimination and selective prediction under a single global rejection threshold, improving micro AUROC from 0.57 to 0.75--0.80. The best results are achieved by a rank-specific multi-head design (H3), suggesting that accounting for hierarchical output structure is important when a unified abstention rule is required. Our code is publicly available at https://github.com/uoguelph-mlrg/hierarchy-aware-llm-uq

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 16 of its 21 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

16 of 21 references.

References · 16

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  5. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  6. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  7. 2023Large Language Models in Finance: A Surveycited by 223
  8. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  9. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  10. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  11. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  12. 2013Evaluation Measures for Hierarchical Classification: a unified view and novel approachescited by 46
4 more
  1. 2023A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Datasetcited by 30
  2. 2024Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generationcited by 16
  3. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  4. 2025Taxonomic Reasoning for Rare Arthropods: Combining Dense Image Captioning and RAG for Interpretable Classificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoning
  2. 2026ChainUQ: Reasoning Consistency-Aware Uncertainty Quantification for Large Language Models
  3. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  4. 2025Towards Label-Free Biological Reasoning Synthetic Dataset Creation via Uncertainty Filtering
  5. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought
  6. 2025LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet?
  7. 2024SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
  8. 2025FishNet++: Analyzing the capabilities of Multimodal Large Language Models in marine biology