# Conformal prediction for text infilling and part-of-speech prediction

arXiv: https://arxiv.org/abs/2111.02592 · HTML: https://mlsanity.com/paper/2111.02592/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 73 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Enabling Language Models to Fill in the Blanks](https://mlsanity.com/paper/2005.05339/) (2020)
- [MaskGAN: Better Text Generation via Filling in the______](https://mlsanity.com/paper/1801.07736/) (2018)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [A Primer on Neural Network Models for Natural Language Processing](https://mlsanity.com/paper/1510.00726/) (2015)
- [A Clockwork RNN](https://mlsanity.com/paper/1402.3511/) (2014)
- [Few-shot Conformal Prediction with Auxiliary Tasks](https://mlsanity.com/paper/2102.08898/) (2021)
- [Blank Language Models](https://mlsanity.com/paper/2002.03079/) (2020)
- [Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss](https://mlsanity.com/paper/1604.05529/) (2016)
- [Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://mlsanity.com/paper/1508.02096/) (2015)
- [Empower Sequence Labeling with Task-Aware Neural Language Model](https://mlsanity.com/paper/1709.04109/) (2017)
- [Robust Multilingual Part-of-Speech Tagging via Adversarial Training](https://mlsanity.com/paper/1711.04903/) (2017)
- [Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings](https://mlsanity.com/paper/1805.08237/) (2018)
- [Overcoming the vanishing gradient problem in plain recurrent networks](https://mlsanity.com/paper/1801.06105/) (2018)
- [A Simple and Effective Approach to the Story Cloze Test](https://mlsanity.com/paper/1803.05547/) (2018)
- [TIGS: An Inference Algorithm for Text Infilling with Gradient Search](https://mlsanity.com/paper/1905.10752/) (2019)
- [Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network](https://mlsanity.com/paper/1510.06168/) (2015)
- [Learning Better Internal Structure of Words for Sequence Labeling](https://mlsanity.com/paper/1810.12443/) (2018)
- [Efficient Conformal Prediction via Cascaded Inference with Expanded Admission](https://mlsanity.com/paper/2007.03114/) (2020)
- [Efficient Set-Valued Prediction in Multi-Class Classification](https://mlsanity.com/paper/1906.08129/) (2019)
- [Reliable Part-of-Speech Tagging of Historical Corpora through Set-Valued Prediction](https://mlsanity.com/paper/2008.01377/) (2020)

## Built on by (most cited first, 22 of 22)

- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access](https://mlsanity.com/paper/2403.01216/) (2024)
- [Conformal Prediction for Natural Language Processing: A Survey](https://mlsanity.com/paper/2405.01976/) (2024)
- [Uncertainty-Aware Evaluation for Vision-Language Models](https://mlsanity.com/paper/2402.14418/) (2024)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [Non-Exchangeable Conformal Language Generation with Nearest Neighbors](https://mlsanity.com/paper/2402.00707/) (2024)
- [CP-Router: An Uncertainty-Aware Router Between LLM and LRM](https://mlsanity.com/paper/2505.19970/) (2025)
- [Conformal Autoregressive Generation: Beam Search with Coverage Guarantees](https://mlsanity.com/paper/2309.03797/) (2023)
- [Evaluating Machine Translation Quality with Conformal Predictive Distributions](https://mlsanity.com/paper/2306.01549/) (2023)
- [Calibration of Natural Language Understanding Models with Venn--ABERS Predictors](https://mlsanity.com/paper/2205.10586/) (2022)
- [Uncertainty in Action: Confidence Elicitation in Embodied Agents](https://mlsanity.com/paper/2503.10628/) (2025)
- [Uncertainty quantification in automated valuation models with spatially weighted conformal prediction](https://mlsanity.com/paper/2312.06531/) (2023)
- [A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement](https://mlsanity.com/paper/2411.04090/) (2024)
- [Conformal Predictor for Improving Zero-shot Text Classification Efficiency](https://mlsanity.com/paper/2210.12619/) (2022)
- [Bin-Conditional Conformal Prediction of Fatalities from Armed Conflict](https://mlsanity.com/paper/2410.14507/) (2024)
- [Another Fit Bites the Dust: Conformal Prediction as a Calibration Standard for Machine Learning in High-Energy Physics](https://mlsanity.com/paper/2512.17048/) (2025)
- [A Confidence Machine for Sparse High-Order Interaction Model](https://mlsanity.com/paper/2205.14317/) (2022)
- [Uncertainty-Aware Transformers: Conformal Prediction for Language Models](https://mlsanity.com/paper/2604.08885/) (2026)

## Abstract (arXiv)

Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data. We construct new conformal prediction-enhanced bidirectional encoder representations from transformers (BERT) and bidirectional long short-term memory (BiLSTM) algorithms for POS tagging and a new conformal prediction-enhanced BERT algorithm for text infilling. We analyze the performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve machine generated audio transcriptions.