mlsanityData to 13 Oct 2026

Reliable Financial Named Entity Recognition Under Domain Shift: Confidence Estimation and Selective Prediction

arXiv 2608.19558 ↗First posted 20 Aug 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, and standard F1 scores do not indicate which predictions remain safe to automate when that input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition, evaluating a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models with five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

30 of 39 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  3. 2017On Calibration of Modern Neural Networkscited by 4,902
  4. 2024Qwen2.5 Technical Reportcited by 4,840
  5. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,371
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,278
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 960
  9. 2023BloombergGPT: A Large Language Model for Financecited by 932
  10. 2017Selective Classification for Deep Neural Networkscited by 825
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 810
  12. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 798
18 more
  1. 2020Calibration of Pre-trained Transformerscited by 308
  2. 2020Consistent Estimators for Learning to Defer to an Expertcited by 268
  3. 2020Selective Question Answering under Domain Shiftcited by 231
  4. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 159
  5. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 141
  6. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  7. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  8. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  9. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 71
  10. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  11. 2021RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Modelscited by 28
  12. 2022Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shiftscited by 28
  13. 2022A Learning Based Hypothesis Test for Harmful Covariate Shiftcited by 18
  14. 2020Calibrating Structured Output Predictors for Natural Language Processingcited by 17
  15. 2023FiNER-ORD: Financial Named Entity Recognition Open Research Datasetcited by 17
  16. 2024Rethinking Negative Instances for Generative Named Entity Recognitioncited by 13
  17. 2025Financial Named Entity Recognition: How Far Can LLM Go?cited by 7
  18. 2026Instruction Finetuning DeepSeek-R1-8B Model Using LoRA and NEFTunecited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Harnessing LLMs as Agents: What Does It Cost?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models
  2. 2026A Training-Free, Alignment-Free Approach to Corporate Intelligence: Application to SEC Filings
  3. 2026FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting
  4. 2025Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications
  5. 2024FANAL -- Financial Activity News Alerting Language Modeling Framework
  6. 2026FinAnchor: Aligned Multi-Model Representations for Financial Prediction
  7. 2026Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training
  8. 2026Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents