# Topical: Learning Repository Embeddings from Source Code using Attention

arXiv: https://arxiv.org/abs/2208.09495 · HTML: https://mlsanity.com/paper/2208.09495/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 19 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [Exploring Software Naturalness through Neural Language Models](https://mlsanity.com/paper/2006.12641/) (2020)
- [Studying Software Engineering Patterns for Designing Machine Learning Systems](https://mlsanity.com/paper/1910.04736/) (2019)
- [Topic Recommendation for Software Repositories using Multi-label Classification Algorithms](https://mlsanity.com/paper/2010.09116/) (2020)
- [HiGitClass: Keyword-Driven Hierarchical Classification of GitHub Repositories](https://mlsanity.com/paper/1910.07115/) (2019)
- [PyCG: Practical Call Graph Generation in Python](https://mlsanity.com/paper/2103.00587/) (2021)
- [SCELMo: Source Code Embeddings from Language Models](https://mlsanity.com/paper/2004.13214/) (2020)
- [Import2vec - Learning Embeddings for Software Libraries](https://mlsanity.com/paper/1904.03990/) (2019)
- [A Dataset for GitHub Repository Deduplication: Extended Description](https://mlsanity.com/paper/2002.02314/) (2020)
- [Repo2Vec: A Comprehensive Embedding Approach for Determining Repository Similarity](https://mlsanity.com/paper/2107.05112/) (2021)

## Abstract (arXiv)

This paper presents Topical, a novel deep neural network for repository level embeddings. Existing methods, reliant on natural language documentation or naive aggregation techniques, are outperformed by Topical's utilization of an attention mechanism. This mechanism generates repository-level representations from source code, full dependency graphs, and script level textual data. Trained on publicly accessible GitHub repositories, Topical surpasses multiple baselines in tasks such as repository auto-tagging, highlighting the attention mechanism's efficacy over traditional aggregation methods. Topical also demonstrates scalability and efficiency, making it a valuable contribution to repository-level representation computation. For further research, the accompanying tools, code, and training dataset are provided at: https://github.com/jpmorganchase/topical.