Language Agnostic Code Embeddings
Not on the map. Why?
Abstract (arXiv)
Recently, code language models have achieved notable advancements in addressing a diverse array of essential code comprehension and generation tasks. Yet, the field lacks a comprehensive deep dive and understanding of the code embeddings of multilingual code models. In this paper, we present a comprehensive study on multilingual code embeddings, focusing on the cross-lingual capabilities of these embeddings across different programming languages. Through probing experiments, we demonstrate that code embeddings comprise two distinct components: one deeply tied to the nuances and syntax of a specific language, and the other remaining agnostic to these details, primarily focusing on semantics. Further, we show that when we isolate and eliminate this language-specific component, we witness significant improvements in downstream code retrieval tasks, leading to an absolute increase of up to +17 in the Mean Reciprocal Rank (MRR).
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 25 Oct 2023.
Builds on
32 of 39 references.
References · 32
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
- 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
- 2023StarCoder: may the source be with you!cited by 790
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
- 2019How multilingual is Multilingual BERT?cited by 642
20 more
- 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
- 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
- 2021Unified Pre-training for Program Understanding and Generationcited by 361
- 2022A Systematic Evaluation of Large Language Models of Codecited by 360
- 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
- 2022The Stack: 3 TB of permissively licensed source codecited by 354
- 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
- 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
- 2023SantaCoder: don't reach for the stars!cited by 169
- 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
- 2020Efficient Domain Generalization via Common-Specific Low-Rank Decompositioncited by 134
- 2022The Geometry of Multilingual Language Model Representationscited by 94
- 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
- 2020Contrastive Code Representation Learningcited by 76
- 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 69
- 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
- 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 54
- 2020LAReQA: Language-agnostic answer retrieval from a multilingual poolcited by 41
- 2020Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Studycited by 24
- 2021A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representationscited by 16
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.