# AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation arXiv: https://arxiv.org/abs/2305.12050 ยท HTML: https://mlsanity.com/paper/2305.12050/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 36 references that are in the map. - First posted on arXiv 20 May 2023. ## Builds on (most cited first, 18 of 18) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019) - [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022) - [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022) - [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022) - [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023) - [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022) - [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020) - [Improving ChatGPT Prompt for Code Generation](https://mlsanity.com/paper/2305.08360/) (2023) - [The Dark Side of ChatGPT: Legal and Ethical Challenges from Stochastic Parrots and Hallucination](https://mlsanity.com/paper/2304.14347/) (2023) - [Improving Code Autocompletion with Transfer Learning](https://mlsanity.com/paper/2105.05991/) (2021) ## Built on by (most cited first, 21 of 21) - [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://mlsanity.com/paper/2312.04724/) (2023) - [Large Language Models for Supply Chain Optimization](https://mlsanity.com/paper/2307.03875/) (2023) - [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024) - [A Cost Analysis of Generative Language Models and Influence Operations](https://mlsanity.com/paper/2308.03740/) (2023) - [Enhancing LLM Performance Through Debate: An Empirical Study on Multi-Agent Debate for Coding Tasks](https://mlsanity.com/paper/2503.12029/) (2025) - [Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models](https://mlsanity.com/paper/2503.12293/) (2025) - [Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback](https://mlsanity.com/paper/2507.18755/) (2025) - [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025) - [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025) - [Double Backdoored: Converting Code Large Language Model Backdoors to Traditional Malware via Adversarial Instruction Tuning Attacks](https://mlsanity.com/paper/2404.18567/) (2024) - [Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directions](https://mlsanity.com/paper/2505.08135/) (2025) - [Optimizing Large Language Models to Expedite the Development of Smart Contracts](https://mlsanity.com/paper/2310.05178/) (2023) - [CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora](https://mlsanity.com/paper/2604.18027/) (2026) - [AI-Assisted Fixes to Code Review Comments at Scale](https://mlsanity.com/paper/2507.13499/) (2025) - [Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Tasks](https://mlsanity.com/paper/2510.11516/) (2025) - [Assistance to Autonomy: A Systematic Literature Review of Agentic AI across the Software Development Life Cycle](https://mlsanity.com/paper/2605.15245/) (2026) - [Echo: Learning from Experience Data via User-Driven Refinement](https://mlsanity.com/paper/2605.21984/) (2026) - [Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice](https://mlsanity.com/paper/2606.22327/) (2026) - [From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data](https://mlsanity.com/paper/2607.16845/) (2026) - [Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning](https://mlsanity.com/paper/2609.17338/) (2026) - [E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch](https://mlsanity.com/paper/2609.38335/) (2026) ## Abstract (arXiv) Generative LLMs have been shown to effectively power AI-based code authoring tools that can suggest entire statements or blocks of code during code authoring. In this paper we present CodeCompose, an AI-assisted code authoring tool developed and deployed at Meta internally. CodeCompose is based on the InCoder LLM that merges generative capabilities with bi-directionality. We have scaled up CodeCompose to serve tens of thousands of developers at Meta, across 9 programming languages and several coding surfaces. We present our experience in making design decisions about the model and system architecture for CodeCompose that addresses these challenges. To release a LLM model at this scale, we needed to first ensure that it is sufficiently accurate. In a random sample of 20K source code files, depending on the language, we are able to reproduce hidden lines between 40% and 58% of the time, an improvement of 1.4x and 4.1x over a model trained only on public data. We gradually rolled CodeCompose out to developers. At the time of this writing, 16K developers have used it with 8% of their code coming directly from CodeCompose. To triangulate our numerical findings, we conduct a thematic analysis on the feedback from 70 developers. We find that 91.5% of the feedback is positive, with the most common themes being discovering APIs, dealing with boilerplate code, and accelerating coding. Meta continues to integrate this feedback into CodeCompose.