# LSem2Vec: A Simple yet Effective Two-Stage Approach for Source Code Embedding

arXiv: https://arxiv.org/abs/2409.14644 · HTML: https://mlsanity.com/paper/2409.14644/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 58 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [code2seq: Generating Sequences from Structured Representations of Code](https://mlsanity.com/paper/1808.01400/) (2018)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020)
- [Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree](https://mlsanity.com/paper/2002.08653/) (2020)
- [SourcererCC: Scaling Code Clone Detection to Big Code](https://mlsanity.com/paper/1512.06448/) (2015)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)
- [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021)
- [Code Representation Learning At Scale](https://mlsanity.com/paper/2402.01935/) (2024)
- [InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees](https://mlsanity.com/paper/2012.07023/) (2020)
- [CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search](https://mlsanity.com/paper/2201.10866/) (2022)
- [GraphSearchNet: Enhancing GNNs via Capturing Global Dependencies for Semantic Code Search](https://mlsanity.com/paper/2111.02671/) (2021)
- [Investigating the Efficacy of Large Language Models for Code Clone Detection](https://mlsanity.com/paper/2401.13802/) (2024)
- [Assured LLM-Based Software Engineering](https://mlsanity.com/paper/2402.04380/) (2024)
- [TransformCode: A Contrastive Learning Framework for Code Embedding via Subtree Transformation](https://mlsanity.com/paper/2311.08157/) (2023)

## Abstract (arXiv)

The advent of large language models (LLMs) has significantly advanced artificial intelligence in software engineering, with source code embeddings playing a crucial role in tasks such as source code clone detection and source code clustering. However, existing methods for source code embedding, including those based on LLMs, often rely on costly supervised training or fine-tuning for domain adaptation. This paper proposes LSem2Vec (LLM-extracted code Semantics to Vector embedding), a simple yet effective two-stage approach to embedding source code by combining large language and sentence embedding models. Specifically, LSem2Vec leverages an LLM to extract the semantics of source code, and then uses a sentence embedding model to generate representation vectors. Compared with previous approaches, LSem2Vec eliminates the need for task-specific training or fine-tuning and effectively addresses erroneous information commonly found in LLM-generated outputs. To evaluate the performance of LSem2Vec, we conducted a series of experiments across three datasets spanning different programming languages, using various LLMs and sentence embedding models. The experimental results demonstrate the effectiveness and superiority of LSem2Vec over five state-of-the-art unsupervised methods. Our findings highlight the potential of LSem2Vec to advance the field of software engineering by providing robust and efficient solutions for source code embedding tasks.