mlsanityData to 7 Oct 2026

The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results

arXiv 2109.00498 ↗First posted 31 Aug 2021cs.LO cs.LGcited by 71

Not on the map. Why?

Abstract (arXiv)

This report summarizes the second International Verification of Neural Networks Competition (VNN-COMP 2021), held as a part of the 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems that was collocated with the 33rd International Conference on Computer-Aided Verification (CAV). Twelve teams participated in this competition. The goal of the competition is to provide an objective comparison of the state-of-the-art methods in neural network verification, in terms of scalability and speed. Along this line, we used standard formats (ONNX for neural networks and VNNLIB for specifications), standard hardware (all tools are run by the organizers on AWS), and tool parameters provided by the tool authors. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this competition.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 28 of its 53 references that are in the map. First posted on arXiv 31 Aug 2021.

Builds on

28 of 53 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  5. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  6. 2018Efficient Neural Network Robustness Certification with General Activation Functionscited by 439
  7. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  8. 2017The Case for Learned Index Structurescited by 188
  9. 2020Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifierscited by 183
  10. 2019Branch and Bound for Piecewise Linear Neural Network Verificationcited by 138
  11. 2020NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systemscited by 128
  12. 2018A Dual Approach to Scalable Verification of Deep Networkscited by 125
16 more
  1. 2019A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networkscited by 124
  2. 2018Strong mixed-integer programming formulations for trained neural networkscited by 115
  3. 2017Output Reachable Set Estimation and Verification for Multi-Layer Neural Networkscited by 79
  4. 2020Verification of Deep Convolutional Neural Networks Using ImageStarscited by 75
  5. 2020An SMT-Based Approach for Verifying Binarized Neural Networkscited by 47
  6. 2019Neural Network Branching for Neural Network Verificationcited by 45
  7. 2021Improved Branch and Bound for Neural Network Verification via Lagrangian Decompositioncited by 44
  8. 2020Scaling Polyhedral Neural Network Verification on GPUscited by 43
  9. 2020Parallelization Techniques for Verifying Neural Networkscited by 36
  10. 2020Reachable Polyhedral Marching (RPM): A Safety Verification Algorithm for Robotic Systems with Deep Neural Network Componentscited by 36
  11. 2020Lagrangian Decomposition for Neural Network Verificationcited by 35
  12. 2021DNNV: A Framework for Deep Neural Network Verificationcited by 23
  13. 2021Generating Adversarial Examples with Graph Neural Networkscited by 12
  14. 2021Scaling the Convex Barrier with Sparse Dual Algorithmscited by 10
  15. 2020Debona: Decoupled Boundary Network Analysis for Tighter Bounds and Faster Adversarial Robustness Proofscited by 8
  16. 2021Neural Network Branch-and-Bound for Neural Network Verificationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 28; 2023: 20; 2024: 6; 2025: 6; 2026: 8.

71 papers.

Sort:

By area:

  1. 2022General Cutting Planes for Bound-Propagation-Based Neural Network Verificationcited by 102
  2. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 64
  3. 2023First Three Years of the International Verification of Neural Networks Competition (VNN-COMP)cited by 58
  4. 2024Scalable Neural Network Verification with Branch-and-bound Inferred Cutting Planescited by 33
  5. 2022An Abstraction-Refinement Approach to Verifying Convolutional Neural Networkscited by 24
  6. 2022Efficient Neural Network Analysis with Sum-of-Infeasibilitiescited by 24
  7. 2022Open- and Closed-Loop Neural Network Verification using Polynomial Zonotopescited by 24
  8. 2023Expressive Losses for Verified Robustness via Convex Combinationscited by 23
  9. 2023A DPLL(T) Framework for Verifying Deep Neural Networkscited by 23
  10. 2022Neural Network Verification with Proof Productioncited by 22
  11. 2022IBP Regularization for Verified Adversarial Robustness via Branch-and-Boundcited by 21
  12. 2023Enumerating Safe Regions in Deep Neural Networks with Provable Probabilistic Guaranteescited by 19
38 more
  1. 2023Logic of Differentiable Logics: Towards a Uniform Semantics of DLcited by 15
  2. 2022Can pruning improve certified robustness of neural networks?cited by 14
  3. 2023Incremental Verification of Neural Networkscited by 14
  4. 2024Harnessing Neuron Stability to Improve DNN Verificationcited by 13
  5. 2025Neural Network Verification is a Programming Language Challengecited by 12
  6. 2022Abstraction and Refinement: Towards Scalable and Exact Verification of Neural Networkscited by 11
  7. 2021Shared Certificates for Neural Network Verificationcited by 10
  8. 2022On Neural Network Equivalence Checking using SMT Solverscited by 10
  9. 2022Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustnesscited by 10
  10. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 10
  11. 2025Clip-and-Verify: Linear Constraint-Driven Domain Clipping for Accelerating Neural Network Verificationcited by 9
  12. 2022Neural Network Compression of ACAS Xu Early Prototype is Unsafe: Closed-Loop Verification through Quantized State Backreachabilitycited by 8
  13. 2022Tighter Abstract Queries in Neural Network Verificationcited by 8
  14. 2023Asymmetric Certified Robustness via Feature-Convex Neural Networkscited by 8
  15. 2022Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networkscited by 7
  16. 2022On Optimizing Back-Substitution Methods for Neural Network Verificationcited by 6
  17. 2022Provable Fairness for Neural Network Models using Formal Verificationcited by 6
  18. 2023Fully Automatic Neural Network Reduction for Formal Verificationcited by 6
  19. 2023ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verificationcited by 6
  20. 2022BERN-NN: Tight Bound Propagation For Neural Networks Using Bernstein Polynomial Interval Arithmeticcited by 5
  21. 2023Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approachcited by 5
  22. 2024SoundnessBench: A Soundness Benchmark for Neural Network Verifierscited by 5
  23. 2022Chordal Sparsity for SDP-based Neural Network Verificationcited by 4
  24. 2022A Scalable, Interpretable, Verifiable & Differentiable Logic Gate Convolutional Neural Network Architecture From Truth Tablescited by 4
  25. 2022Global Performance Guarantees for Neural Network Models of AC Power Flowcited by 4
  26. 2022Optimized Symbolic Interval Propagation for Neural Network Verificationcited by 4
  27. 2023GPU-Accelerated Verification of Machine Learning Models for Power Systemscited by 4
  28. 2021Reachability analysis of neural networks using mixed monotonicitycited by 3
  29. 2023A Robust Optimisation Perspective on Counterexample-Guided Repair of Neural Networkscited by 3
  30. 2023DelBugV: Delta-Debugging Neural Network Verifierscited by 3
  31. 2025PICID: Proof-Driven Clause Learning in Neural Network Verificationcited by 3
  32. 2022VPN: Verification of Poisoning in Neural Networkscited by 2
  33. 2022CEG4N: Counter-Example Guided Neural Network Quantization Refinementcited by 2
  34. 2026Incremental Neural Network Verification via Learned Conflictscited by 2
  35. 2022Verifying Inverse Model Neural Networkscited by 1
  36. 2022Differentiable Logics for Neural Network Training and Verificationcited by 1
  37. 2022An Overview and Prospective Outlook on Robust Training and Certification of Machine Learning Modelscited by 1
  38. 2022Tightening Robustness Verification of MaxPool-based Neural Networks via Minimizing the Over-Approximation Zonecited by 1

50 most cited of 71 All 71 (JSON)