mlsanityData to 7 Oct 2026

UniXcoder: Unified Cross-Modal Pre-training for Code Representation

arXiv 2203.03850 ↗First posted 8 Mar 2022cs.CL cs.PL cs.SEcited by 355

Not on the map. Why?

Cited by 32 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained models for programming languages have recently demonstrated great success on code intelligence. To support both code-related understanding and generation tasks, recent works attempt to pre-train unified encoder-decoder models. However, such encoder-decoder framework is sub-optimal for auto-regressive tasks, especially code completion that requires a decoder-only manner for efficient inference. In this paper, we present UniXcoder, a unified cross-modal pre-trained model for programming language. The model utilizes mask attention matrices with prefix adapters to control the behavior of the model and leverages cross-modal contents like AST and code comment to enhance code representation. To encode AST that is represented as a tree in parallel, we propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task. We evaluate UniXcoder on five code-related tasks over nine datasets. To further evaluate the performance of code fragment representation, we also construct a dataset for a new task, called zero-shot code-to-code search. Results show that our model achieves state-of-the-art performance on most tasks and analysis reveals that comment and AST can both enhance UniXcoder.

Key facts. Cited by 355 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 8 Mar 2022.

Builds on

23 of 37 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
11 more
  1. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  2. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  3. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  4. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  5. 2018Mapping Language to Code in Programmatic Contextcited by 104
  6. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  7. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  8. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  9. 2021TreeBERT: A Tree-Based Pre-Trained Model for Programming Languagecited by 40
  10. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  11. 2020Exploring Software Naturalness through Neural Language Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 27; 2023: 71; 2024: 84; 2025: 97; 2026: 76.

355 papers.

Sort:

By area:

  1. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  4. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  5. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  6. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  7. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  8. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  9. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  10. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  11. 2023ChipGPT: How far are we from natural language hardware designcited by 101
  12. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
38 more
  1. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  4. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
  5. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  6. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  7. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  8. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  9. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  10. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  11. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  12. 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
  13. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  14. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 45
  15. 2025LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insightscited by 44
  16. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  17. 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
  18. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 40
  19. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  20. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  21. 2023Automatic Code Summarization via ChatGPT: How Far Are We?cited by 34
  22. 2023Code Execution with Pre-trained Language Modelscited by 33
  23. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  24. 2024REPOFUSE: Repository-Level Code Completion with Fused Dual Contextcited by 32
  25. 2024An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detectioncited by 32
  26. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  27. 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
  28. 2024Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuningcited by 30
  29. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  30. 2023Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attackscited by 29
  31. 2023CodeKGC: Code Language Model for Generative Knowledge Graph Constructioncited by 28
  32. 2024SemCoder: Training Code Language Models with Comprehensive Semantics Reasoningcited by 27
  33. 2024Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detectioncited by 27
  34. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  35. 2024Code Representation Learning At Scalecited by 26
  36. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  37. 2024Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Modelscited by 22
  38. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyondcited by 22

50 most cited of 355 All 355 (JSON)