# PIE: Performance Interval Estimation for Free-Form Generation Tasks

arXiv: https://arxiv.org/abs/2509.07309 · HTML: https://mlsanity.com/paper/2509.07309/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 73 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Builds on (most cited first, 30 of 47)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://mlsanity.com/paper/2502.05167/) (2025)
- [Know Your Limits: A Survey of Abstention in Large Language Models](https://mlsanity.com/paper/2407.18418/) (2024)
- [What Large Language Models Know and What People Think They Know](https://mlsanity.com/paper/2401.13835/) (2024)

## Built on by (most cited first, 1 of 1)

- [RELISH: LLM REgression with a Latent Iterative State Head](https://mlsanity.com/paper/2604.01206/) (2026)

## Abstract (arXiv)

Confidence estimation infers a probability for whether each model output is correct or not. While predicting such binary correctness is sensible for tasks with exact answers, free-form generation tasks are often more nuanced, with output quality being both fine-grained and multi-faceted. We thus propose Performance Interval Estimation (PIE) to predict both: 1) point estimates for any arbitrary set of continuous-valued evaluation metrics; and 2) calibrated uncertainty intervals around these point estimates. We then compare two approaches: LLM-as-judge vs. classic regression with confidence estimation features. Evaluation over 11 datasets spans summarization, translation, code generation, function-calling, and question answering. Regression is seen to achieve both: i) lower error point estimates of metric scores; and ii) well-calibrated uncertainty intervals. To support reproduction and follow-on work, we share our data and code.