# TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills arXiv: https://arxiv.org/abs/2306.07285 ยท HTML: https://mlsanity.com/paper/2306.07285/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. - Builds on 43 of its 67 references that are in the map. - First posted on arXiv 23 May 2023. ## Placed in - Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/) ## Builds on (most cited first, 30 of 43) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016) - [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019) - [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020) - [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021) - [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023) - [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019) - [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020) - [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021) - [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019) - [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022) - [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020) - [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017) ## Built on by (most cited first, 8 of 8) - [CodeFusion: A Pre-trained Diffusion Model for Code Generation](https://mlsanity.com/paper/2310.17680/) (2023) - [CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation](https://mlsanity.com/paper/2504.15254/) (2025) - [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025) - [JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence](https://mlsanity.com/paper/2510.23538/) (2025) - [EffiReasonTrans: RL-Optimized Reasoning for Code Translation](https://mlsanity.com/paper/2510.18863/) (2025) - [Structure-aware Fine-tuning for Code Pre-trained Models](https://mlsanity.com/paper/2404.07471/) (2024) - [SoftSRV: Learn to Generate Targeted Synthetic Data](https://mlsanity.com/paper/2410.16534/) (2024) - [Tabularis Formatus: Predictive Formatting for Tables](https://mlsanity.com/paper/2508.11121/) (2025) ## Abstract (arXiv) Code pre-trained models (CodePTMs) have recently demonstrated a solid capacity to process various software intelligence tasks, e.g., code clone detection, code translation, and code summarization. The current mainstream method that deploys these models to downstream tasks is to fine-tune them on individual tasks, which is generally costly and needs sufficient data for large models. To tackle the issue, in this paper, we present TransCoder, a unified Transferable fine-tuning strategy for Code representation learning. Inspired by human inherent skills of knowledge generalization, TransCoder drives the model to learn better code-related meta-knowledge like human programmers. Specifically, we employ a tunable prefix encoder as the meta-learner to capture cross-task and cross-language transferable knowledge, respectively. Besides, tasks with minor training sample sizes and languages with small corpus can be remarkably benefited from our approach. Extensive experiments conducted on benchmark datasets clearly demonstrate that our method can lead to superior performance on various code-related tasks and encourage mutual reinforcement. We also show that TransCoder is applicable in low-resource scenarios. Our codes are available at https://github.com/QiushiSun/TransCoder.