# Code Compliance Assessment as a Learning Problem

arXiv: https://arxiv.org/abs/2209.04602 · HTML: https://mlsanity.com/paper/2209.04602/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 10 Sep 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [DeepBugs: A Learning Approach to Name-based Bug Detection](https://mlsanity.com/paper/1805.11683/) (2018)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [Unsupervised Pretraining for Sequence to Sequence Learning](https://mlsanity.com/paper/1611.02683/) (2016)
- [When Deep Learning Met Code Search](https://mlsanity.com/paper/1905.03813/) (2019)
- [Multi-Modal Attention Network Learning for Semantic Source Code Retrieval](https://mlsanity.com/paper/1909.13516/) (2019)
- [CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning](https://mlsanity.com/paper/1904.00720/) (2019)
- [Deep Graph Matching and Searching for Semantic Code Retrieval](https://mlsanity.com/paper/2010.12908/) (2020)
- [CORE: Automating Review Recommendation for Code Changes](https://mlsanity.com/paper/1912.09652/) (2019)
- [SWIM: Synthesizing What I Mean](https://mlsanity.com/paper/1511.08497/) (2015)
- [Leveraging Unsupervised Learning to Summarize APIs Discussed in Stack Overflow](https://mlsanity.com/paper/2111.13962/) (2021)
- [Neural Bug Finding: A Study of Opportunities and Challenges](https://mlsanity.com/paper/1906.00307/) (2019)
- [CoNCRA: A Convolutional Neural Network Code Retrieval Approach](https://mlsanity.com/paper/2009.01959/) (2020)
- [What Do Developers Discuss about Code Comments?](https://mlsanity.com/paper/2108.07648/) (2021)

## Abstract (arXiv)

Manual code reviews and static code analyzers are the traditional mechanisms to verify if source code complies with coding policies. However, these mechanisms are hard to scale. We formulate code compliance assessment as a machine learning (ML) problem, to take as input a natural language policy and code, and generate a prediction on the code's compliance, non-compliance, or irrelevance. This can help scale compliance classification and search for policies not covered by traditional mechanisms. We explore key research questions on ML model formulation, training data, and evaluation setup. The core idea is to obtain a joint code-text embedding space which preserves compliance relationships via the vector distance of code and policy embeddings. As there is no task-specific data, we re-interpret and filter commonly available software datasets with additional pre-training and pre-finetuning tasks that reduce the semantic gap. We benchmarked our approach on two listings of coding policies (CWE and CBP). This is a zero-shot evaluation as none of the policies occur in the training set. On CWE and CBP respectively, our tool Policy2Code achieves classification accuracies of (59%, 71%) and search MRR of (0.05, 0.21) compared to CodeBERT with classification accuracies of (37%, 54%) and MRR of (0.02, 0.02). In a user study, 24% Policy2Code detections were accepted compared to 7% for CodeBERT.