# GitHub Considered Harmful? Analyzing Open-Source Projects for the Automatic Generation of Cryptographic API Call Sequences

arXiv: https://arxiv.org/abs/2211.13498 · HTML: https://mlsanity.com/paper/2211.13498/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016)
- [Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security](https://mlsanity.com/paper/1710.03135/) (2017)
- [The Impact of Developer Experience in Using Java Cryptography](https://mlsanity.com/paper/1908.01489/) (2019)
- [Java Cryptography Uses in the Wild](https://mlsanity.com/paper/2009.01101/) (2020)

## Built on by (most cited first, 2 of 2)

- [LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations](https://mlsanity.com/paper/2303.09384/) (2023)
- [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://mlsanity.com/paper/2407.07064/) (2024)

## Abstract (arXiv)

GitHub is a popular data repository for code examples. It is being continuously used to train several AI-based tools to automatically generate code. However, the effectiveness of such tools in correctly demonstrating the usage of cryptographic APIs has not been thoroughly assessed. In this paper, we investigate the extent and severity of misuses, specifically caused by incorrect cryptographic API call sequences in GitHub. We also analyze the suitability of GitHub data to train a learning-based model to generate correct cryptographic API call sequences. For this, we manually extracted and analyzed the call sequences from GitHub. Using this data, we augmented an existing learning-based model called DeepAPI to create two security-specific models that generate cryptographic API call sequences for a given natural language (NL) description. Our results indicate that it is imperative to not neglect the misuses in API call sequences while using data sources like GitHub, to train models that generate code.