mlsanityData to 7 Oct 2026

Select, Label, Evaluate: Active Testing in NLP

arXiv 2603.21840 ↗First posted 23 Mar 2026cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Human annotation cost and time remain significant bottlenecks in Natural Language Processing (NLP), with test data annotation being particularly expensive due to the stringent requirement for low-error and high-quality labels necessary for reliable model evaluation. Traditional approaches require annotating entire test sets, leading to substantial resource requirements. Active Testing is a framework that selects the most informative test samples for annotation. Given a labeling budget, it aims to choose the subset that best estimates model performance while minimizing cost and human effort. In this work, we formalize Active Testing in NLP and we conduct an extensive benchmarking of existing approaches across 18 datasets and 4 embedding strategies spanning 4 different NLP tasks. The experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%. Our analysis reveals variations in method effectiveness across different data characteristics and task types, with no single approach emerging as universally superior. Lastly, to address the limitation of requiring a predefined annotation budget in existing sample selection strategies, we introduce an adaptive stopping criterion that automatically determines the optimal number of samples. We release our code at https://github.com/amazon-science/NLPActiveTesting.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

15 of 25 references.

References · 15

  1. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  2. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  3. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  4. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  5. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
  6. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  7. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 33
  8. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  9. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  10. 2018Active Testing: An Efficient and Robust Framework for Estimating Accuracycited by 15
  11. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  12. 2023Zero-Shot Text Classification via Self-Supervised Tuningcited by 6
3 more
  1. 2025Active Few-Shot Learning for Text Classificationcited by 6
  2. 2024Dynamic Strategy Planning for Efficient Question Answering with Large Language Modelscited by 3
  3. 2025No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Successcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026EarlyEval: Cheaper Agent Evaluation via Early Outcome Predictioncited by 2
  2. 2026QUORUM: QUality-Optimized Routing Using Multiple annotatorscited by 0