Predicting generalization performance with correctness discriminators
Not on the map. Why?
Abstract (arXiv)
The ability to predict an NLP model's accuracy on unseen, potentially out-of-distribution data is a prerequisite for trustworthiness. We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data. We achieve this by training a discriminator which predicts whether the output of a given sequence-to-sequence model is correct or not. We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds, and that these bounds are remarkably close together.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 15 Nov 2023.
Builds on
31 of 43 references.
References · 31
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2022Generative Adversarial Networkscited by 5,603
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
19 more
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
- 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
- 2020Calibration of Pre-trained Transformerscited by 307
- 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
- 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
- 2020Selective Question Answering under Domain Shiftcited by 228
- 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
- 2021Assessing Generalization of SGD via Disagreementcited by 116
- 2021Predicting with Confidence on Unseen Distributionscited by 116
- 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
- 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
- 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
- 2018Confidence Modeling for Neural Semantic Parsingcited by 33
- 2022Structural generalization is hard for sequence-to-sequence modelscited by 24
- 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
- 2021Improving Compositional Generalization in Classification Tasks via Structure Annotationscited by 12
- 2023Error Detection for Text-to-SQL Semantic Parsingcited by 11
- 2023Estimating Large Language Model Capabilities without Labeled Test Datacited by 9
- 2019Automatic Accuracy Prediction for AMR Parsingcited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.