# SCELMo: Source Code Embeddings from Language Models

arXiv: https://arxiv.org/abs/2004.13214 · HTML: https://mlsanity.com/paper/2004.13214/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2020.

## Built on by (most cited first, 16 of 16)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages](https://mlsanity.com/paper/2204.09653/) (2022)
- [CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search](https://mlsanity.com/paper/2201.10866/) (2022)
- [Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code](https://mlsanity.com/paper/2205.11739/) (2022)
- [An Exploratory Study on Code Attention in BERT](https://mlsanity.com/paper/2204.10200/) (2022)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models](https://mlsanity.com/paper/2104.01642/) (2021)
- [LExecutor: Learning-Guided Execution](https://mlsanity.com/paper/2302.02343/) (2023)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees](https://mlsanity.com/paper/2201.07984/) (2022)
- [LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognition](https://mlsanity.com/paper/2204.09654/) (2022)
- [Call Me Maybe: Enhancing JavaScript Call Graph Construction using Graph Neural Networks](https://mlsanity.com/paper/2506.18191/) (2025)
- [DeepMutants: Training neural bug detectors with contextual mutations](https://mlsanity.com/paper/2107.06657/) (2021)
- [Understanding Long Programming Languages with Structure-Aware Sparse Attention](https://mlsanity.com/paper/2205.13730/) (2022)
- [Topical: Learning Repository Embeddings from Source Code using Attention](https://mlsanity.com/paper/2208.09495/) (2022)
- [LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models](https://mlsanity.com/paper/2505.14759/) (2025)

## Abstract (arXiv)

Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Contextual embeddings are common in natural language processing but have not been previously applied in software engineering. We introduce a new set of deep contextualized word representations for computer programs based on language models. We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018). We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection. We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.