# Efficient Code Embeddings from Code Generation Models

arXiv: https://arxiv.org/abs/2508.21290 · HTML: https://mlsanity.com/paper/2508.21290/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 29 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [Gemini Embedding: Generalizable Embeddings from Gemini](https://mlsanity.com/paper/2503.07891/) (2025)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [CoIR: A Comprehensive Benchmark for Code Information Retrieval Models](https://mlsanity.com/paper/2407.02883/) (2024)

## Built on by (most cited first, 13 of 13)

- [R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training](https://mlsanity.com/paper/2602.13103/) (2026)
- [CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding](https://mlsanity.com/paper/2606.11864/) (2026)
- [SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generation](https://mlsanity.com/paper/2603.08719/) (2026)
- [MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2602.07848/) (2026)
- [Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents](https://mlsanity.com/paper/2607.24882/) (2026)
- [MARS2: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2604.14564/) (2026)
- [Beyond Retrieval: A Multitask Benchmark and Model for Code Search](https://mlsanity.com/paper/2605.04615/) (2026)
- [Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation](https://mlsanity.com/paper/2605.28022/) (2026)
- [NeSy-RAG: Neuro-Symbolic RAG for Explainable Question Answering](https://mlsanity.com/paper/2608.06292/) (2026)
- [CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents](https://mlsanity.com/paper/2607.25431/) (2026)
- [Stop Indexing at Full Precision: Revisiting Clustering for Vector Embeddings](https://mlsanity.com/paper/2608.14648/) (2026)
- [Invariant Pretraining for Robust Code Representations](https://mlsanity.com/paper/2608.15412/) (2026)
- [Questioning the Questions: Sustaining Self-Evolution in Reasoning Models](https://mlsanity.com/paper/2610.04299/) (2026)

## Abstract (arXiv)

jina-code-embeddings is a novel code embedding model suite designed to retrieve code from natural language queries, perform technical question-answering, and identify semantically similar code snippets across programming languages. It makes innovative use of an autoregressive backbone pre-trained on both text and code, generating embeddings via last-token pooling. We outline the training recipe and demonstrate state-of-the-art performance despite the relatively small size of the models, validating this approach to code embedding model construction.