# Are Machine Programming Systems using Right Source-Code Measures to Select Code Repositories?

arXiv: https://arxiv.org/abs/2209.11946 · HTML: https://mlsanity.com/paper/2209.11946/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 24 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Graph-based, Self-Supervised Program Repair from Diagnostic Feedback](https://mlsanity.com/paper/2005.10636/) (2020)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [Aroma: Code Recommendation via Structural Code Search](https://mlsanity.com/paper/1812.01158/) (2018)
- [What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform](https://mlsanity.com/paper/1811.07643/) (2018)
- [The Three Pillars of Machine Programming](https://mlsanity.com/paper/1803.07244/) (2018)
- [Predicting the Popularity of GitHub Repositories](https://mlsanity.com/paper/1607.04342/) (2016)
- [ControlFlag: A Self-Supervised Idiosyncratic Pattern Detection System for Software Control Structures](https://mlsanity.com/paper/2011.03616/) (2020)
- [GitRank: A Framework to Rank GitHub Repositories](https://mlsanity.com/paper/2205.02360/) (2022)

## Built on by (most cited first, 1 of 1)

- [CompCodeVet: A Compiler-guided Validation and Enhancement Approach for Code Dataset](https://mlsanity.com/paper/2311.06505/) (2023)

## Abstract (arXiv)

Machine programming (MP) is an emerging field at the intersection of deterministic and probabilistic computing, and it aims to assist software and hardware engineers, among other applications. Along with powerful compute resources, MP systems often rely on vast amount of open-source code to learn interesting properties about code and programming and solve problems in the areas of debugging, code recommendation, auto-completion, etc. Unfortunately, several of the existing MP systems either do not consider quality of code repositories or use atypical quality measures than those typically used in software engineering community to select them. As such, impact of quality of code repositories on the performance of these systems needs to be studied. In this preliminary paper, we evaluate impact of different quality repositories on the performance of a candidate MP system. Towards that objective, we develop a framework, named GitRank, to rank open-source repositories on quality, maintainability, and popularity by leveraging existing research on this topic. We then apply GitRank to evaluate correlation between the quality measures used by the candidate MP system and the quality measures used by our framework. Our preliminary results reveal some correlation between the quality measures used in GitRank and ControlFlag's performance, suggesting that some of the measures used in GitRank are applicable to ControlFlag. But it also raises questions around right quality measures for code repositories used in MP systems. We believe that our findings also generate interesting insights towards code quality measures that affect performance of MP systems.