mlsanityData to 7 Oct 2026

Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models

arXiv 2410.24005 ↗First posted 31 Oct 2024cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

The predominant de facto paradigm of testing ML models relies on either using only held-out data to compute aggregate evaluation metrics or by assessing the performance on different subgroups. However, such data-only testing methods operate under the restrictive assumption that the available empirical data is the sole input for testing ML models, disregarding valuable contextual information that could guide model testing. In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism. Through empirical evaluations in diverse settings, we show that SMART automatically identifies more relevant and impactful failures than alternatives, demonstrating the potential of CAT as a testing paradigm.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 26 of its 56 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

26 of 56 references.

References · 26

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2023Large Language Models as Optimizerscited by 830
  7. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  8. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  9. 2018Multiaccuracy: Black-Box Post-Processing for Fairness in Classificationcited by 221
  10. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  11. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  12. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
14 more
  1. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  2. 2020Model Patching: Closing the Subgroup Performance Gap with Data Augmentationcited by 82
  3. 2018Automated Data Slicing for Model Validation:A Big data - AI Integration Approachcited by 70
  4. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  5. 2020Understanding Failures of Deep Networks via Robust Feature Extractioncited by 56
  6. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  7. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  8. 2023Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmarkcited by 23
  9. 2018Learning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICUcited by 21
  10. 2022DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systemscited by 14
  11. 2021Thinking Beyond Distributions in Testing Machine Learned Modelscited by 5
  12. 2022Specifying and Testing k-Safety Properties for Machine-Learning Modelscited by 5
  13. 2022Interpreting Black-box Machine Learning Models for High Dimensional Datasetscited by 4
  14. 2018Contingency Trainingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 2.

8 papers.

  1. 2024Quantifying perturbation impacts for large language modelscited by 8
  2. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  3. 2025Deep Hierarchical Learning with Nested Subspace Networks for Large Language Modelscited by 5
  4. 2025On the Origins of Sampling Bias: Implications on Fairness Measurement and Mitigationcited by 1
  5. 2026No More, No Less: Least-Privilege Language Modelscited by 1
  6. 2026Multi-Agent Systems Should be Treated as Principal-Agent Problemscited by 1
  7. 2025"Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Driftcited by 0
  8. 2025What's the next frontier for Data-centric AI? Data Savvy Agentscited by 0