# Improving Code Example Recommendations on Informal Documentation Using BERT and Query-Aware LSH: A Comparative Study

arXiv: https://arxiv.org/abs/2305.03017 · HTML: https://mlsanity.com/paper/2305.03017/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 38 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep API Learning](https://mlsanity.com/paper/1605.08535/) (2016)
- [Leveraging Unsupervised Learning to Summarize APIs Discussed in Stack Overflow](https://mlsanity.com/paper/2111.13962/) (2021)
- [Development of recommendation systems for software engineering: the CROSSMINER experience](https://mlsanity.com/paper/2103.06987/) (2021)
- [Towards Code Summarization of APIs Based on Unofficial Documentation Using NLP Techniques](https://mlsanity.com/paper/2208.06318/) (2022)
- [Senatus -- A Fast and Accurate Code-to-Code Recommendation Engine](https://mlsanity.com/paper/2111.04473/) (2021)

## Built on by (most cited first, 7 of 7)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [Enhancing Recommendation Diversity by Re-ranking with Large Language Models](https://mlsanity.com/paper/2401.11506/) (2024)
- [Large Language Models Make Sample-Efficient Recommender Systems](https://mlsanity.com/paper/2406.02368/) (2024)
- [Automating API Documentation with LLMs: A BERTopic Approach](https://mlsanity.com/paper/2509.05749/) (2025)

## Abstract (arXiv)

Our research investigates the recommendation of code examples to aid software developers, a practice that saves developers significant time by providing ready-to-use code snippets. The focus of our study is Stack Overflow, a commonly used resource for coding discussions and solutions, particularly in the context of the Java programming language. We applied BERT, a powerful Large Language Model (LLM) that enables us to transform code examples into numerical vectors by extracting their semantic information. Once these numerical representations are prepared, we identify Approximate Nearest Neighbors (ANN) using Locality-Sensitive Hashing (LSH). Our research employed two variants of LSH: Random Hyperplane-based LSH and Query-Aware LSH. We rigorously compared these two approaches across four parameters: HitRate, Mean Reciprocal Rank (MRR), Average Execution Time, and Relevance. Our study revealed that the Query-Aware (QA) approach showed superior performance over the Random Hyperplane-based (RH) method. Specifically, it exhibited a notable improvement of 20% to 35% in HitRate for query pairs compared to the RH approach. Furthermore, the QA approach proved significantly more time-efficient, with its speed in creating hashing tables and assigning data samples to buckets being at least four times faster. It can return code examples within milliseconds, whereas the RH approach typically requires several seconds to recommend code examples. Due to the superior performance of the QA approach, we tested it against PostFinder and FaCoY, the state-of-the-art baselines. Our QA method showed comparable efficiency proving its potential for effective code recommendation.