mlsanityData to 7 Oct 2026

ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verification

arXiv 2305.04003 ↗First posted 6 May 2023cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Verification of machine learning models used in Natural Language Processing (NLP) is known to be a hard problem. In particular, many known neural network verification methods that work for computer vision and other numeric datasets do not work for NLP. Here, we study technical reasons that underlie this problem. Based on this analysis, we propose practical methods and heuristics for preparing NLP datasets and models in a way that renders them amenable to known verification methods based on abstract interpretation. We implement these methods as a Python library called ANTONIO that links to the neural network verifiers ERAN and Marabou. We perform evaluation of the tool using an NLP dataset R-U-A-Robot suggested as a benchmark for verifying legally critical NLP applications. We hope that, thanks to its general applicability, this work will open novel possibilities for including NLP verification problems into neural network verification competitions, and will popularise NLP problems within this community.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 6 May 2023.

Builds on

32 of 53 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  10. 2018Robustness May Be at Odds with Accuracycited by 873
  11. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  12. 2021Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verificationcited by 224
20 more
  1. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  2. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  3. 2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 204
  4. 2019Adversarial Training Can Hurt Generalizationcited by 131
  5. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  6. 2019Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagationcited by 87
  7. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  8. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  9. 2020Robustness Verification for Transformerscited by 76
  10. 2020SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutionscited by 72
  11. 2021The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Resultscited by 71
  12. 2021Towards Robustness Against Natural Language Word Substitutionscited by 68
  13. 2021Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitutioncited by 55
  14. 2022The Third International Verification of Neural Networks Competition (VNN-COMP 2022): Summary and Resultscited by 39
  15. 2021Neural Network Robustness as a Verification Property: A Principled Case Studycited by 35
  16. 2020Scalable Quantitative Verification For Deep Neural Networkscited by 31
  17. 2021Certified Robustness to Programmable Transformations in LSTMscited by 18
  18. 2021The R-U-A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identitycited by 16
  19. 2023Logic of Differentiable Logics: Towards a Uniform Semantics of DLcited by 15
  20. 2021InvBERT: Reconstructing Text from Contextualized Word Embeddings by inverting the BERT pipelinecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 1.

6 papers.

  1. 2023Mirages: On Anthropomorphism in Dialogue Systemscited by 30
  2. 2024NLP Verification: Towards a General Methodology for Certifying Robustnesscited by 14
  3. 2025Proof Minimization in Neural Network Verificationcited by 4
  4. 2025PICID: Proof-Driven Clause Learning in Neural Network Verificationcited by 3
  5. 2025A General Framework for Property-Driven Machine Learningcited by 1
  6. 2026Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinementcited by 0