mlsanityData to 14 Oct 2026

Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder

arXiv 2608.19957 ↗First posted 20 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain-specific terminology, for which open datasets and specialized models have been virtually non-existent. We present a comprehensive pipeline for 1C code retrieval: an open benchmark of 3,413 real-world, PII-scrubbed query-code pairs, a reproducible evaluation harness, and a specialized bi-encoder. To overcome scarce labeled data, we fine-tune on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories, using Matryoshka Representation Learning (MRL) and a privacy-aware tokenizer. Because the benchmark subsets differ in size, we report balanced-subset macro, query-weighted micro, and forum-only results. Our model reaches 0.5992 balanced macro nDCG@10, 0.5044 micro, and 0.4617 on forum, versus 0.4932 macro for the baseline architecture and 0.5404 for google/embeddinggemma-300m. Removing every benchmark example flagged by the conservative exact/13-gram overlap audit leaves 0.6011 balanced macro (0.5010 micro), indicating that detected train-benchmark overlap does not explain the headline result. MRL truncation to 256 dimensions preserves 99.9% of retrieval quality while reducing dense-index storage and exact similarity arithmetic by a factor of three.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

20 of 33 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  9. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  11. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  12. 2022Matryoshka Representation Learningcited by 282
8 more
  1. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  2. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 75
  3. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 60
  4. 2023Rethinking Negative Pairs in Code Searchcited by 14
  5. 2025Bridging Language Gaps: Advances in Cross-Lingual Information Retrieval with Multilingual LLMscited by 6
  6. 2025Multilingual Generative Retrieval via Cross-lingual Semantic Compressioncited by 4
  7. 2025Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generationcited by 3
  8. 2026"Write in English, Nobody Understands Your Language Here": A Study of Non-English Trends in Open-Source Repositoriescited by 2

Built on by

Not cited yet.