# LEGION: Harnessing Pre-trained Language Models for GitHub Topic Recommendations with Distribution-Balance Loss

arXiv: https://arxiv.org/abs/2403.05873 · HTML: https://mlsanity.com/paper/2403.05873/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 62 references that are in the map.
- First posted on arXiv 9 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.05830/) (2015)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification](https://mlsanity.com/paper/1909.04176/) (2019)
- [Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution](https://mlsanity.com/paper/2109.04712/) (2021)
- [Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding](https://mlsanity.com/paper/2112.02268/) (2021)
- [Topic Recommendation for Software Repositories using Multi-label Classification Algorithms](https://mlsanity.com/paper/2010.09116/) (2020)
- [AutoPruner: Transformer-Based Call Graph Pruning](https://mlsanity.com/paper/2209.03230/) (2022)
- [Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning](https://mlsanity.com/paper/2301.01113/) (2023)
- [CHRONOS: Time-Aware Zero-Shot Identification of Libraries from Vulnerability Reports](https://mlsanity.com/paper/2301.03944/) (2023)
- [Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial Code](https://mlsanity.com/paper/2208.05361/) (2022)
- [Beware of the Unexpected: Bimodal Taint Analysis](https://mlsanity.com/paper/2301.10545/) (2023)
- [Are We Ready to Embrace Generative AI for Software Q&A?](https://mlsanity.com/paper/2307.09765/) (2023)
- [Evaluating Transfer Learning for Simplifying GitHub READMEs](https://mlsanity.com/paper/2308.09940/) (2023)

## Built on by (most cited first, 1 of 1)

- [DRAGON: Robust Classification for Very Large Collections of Software Repositories](https://mlsanity.com/paper/2602.09071/) (2026)

## Abstract (arXiv)

Open-source development has revolutionized the software industry by promoting collaboration, transparency, and community-driven innovation. Today, a vast amount of various kinds of open-source software, which form networks of repositories, is often hosted on GitHub - a popular software development platform. To enhance the discoverability of the repository networks, i.e., groups of similar repositories, GitHub introduced repository topics in 2017 that enable users to more easily explore relevant projects by type, technology, and more. It is thus crucial to accurately assign topics for each GitHub repository. Current methods for automatic topic recommendation rely heavily on TF-IDF for encoding textual data, presenting challenges in understanding semantic nuances. This paper addresses the limitations of existing techniques by proposing Legion, a novel approach that leverages Pre-trained Language Models (PTMs) for recommending topics for GitHub repositories. The key novelty of Legion is three-fold. First, Legion leverages the extensive capabilities of PTMs in language understanding to capture contextual information and semantic meaning in GitHub repositories. Second, Legion overcomes the challenge of long-tailed distribution, which results in a bias toward popular topics in PTMs, by proposing a Distribution-Balanced Loss (DB Loss) to better train the PTMs. Third, Legion employs a filter to eliminate vague recommendations, thereby improving the precision of PTMs. Our empirical evaluation on a benchmark dataset of real-world GitHub repositories shows that Legion can improve vanilla PTMs by up to 26% on recommending GitHubs topics. Legion also can suggest GitHub topics more precisely and effectively than the state-of-the-art baseline with an average improvement of 20% and 5% in terms of Precision and F1-score, respectively.