mlsanityData to 13 Oct 2026

IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators

arXiv 2403.03894 ↗First posted 6 Mar 2024cs.AI cs.CL cs.PLcited by 21

Not on the map. Why?

Abstract (arXiv)

Code understanding and generation have fast become some of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs (i.e., LMs for code generation) such as cross-lingual transfer between different programming languages, language-specific data augmentation, and post-hoc LM adaptation, alongside exploitation of data sources other than the original textual content, has been much sparser than for their natural language counterparts. In particular, most mainstream Code-LMs have been pre-trained on source code files alone. In this work, we investigate the prospect of leveraging readily available compiler intermediate representations (IR) - shared across programming languages - to improve the multilingual capabilities of Code-LMs and facilitate cross-lingual transfer. To this end, we first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files coupled with respective intermediate representations. Next, starting from various base Code-LMs (ranging in size from 1.1B to 7.3B parameters), we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to (1) learn the IR language and (2) align the IR constructs with respective constructs of various programming languages. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics, including prompt robustness, multilingual code completion, code understanding, and instruction following.

Key facts. Cited by 21 papers in the map (data through 13 Oct 2026). Builds on 58 of its 89 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

58 of 89 references.

References · 58

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  5. 2021Program Synthesis with Large Language Modelscited by 3,573
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,173
  9. 2023Teaching Large Language Models to Self-Debugcited by 972
  10. 2021Measuring Coding Challenge Competence With APPScited by 891
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 884
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 764
38 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 743
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 736
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 717
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 710
  5. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  6. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 360
  7. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 298
  8. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 254
  9. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  10. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  11. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  12. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 153
  13. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 150
  14. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  15. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  16. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 135
  17. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 125
  18. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 112
  19. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 102
  20. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  21. 2020Contrastive Code Representation Learningcited by 76
  22. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  23. 2018Generative Code Modeling with Graphscited by 60
  24. 2019TreeGen: A Tree-Based Transformer Architecture for Code Generationcited by 58
  25. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 54
  26. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 47
  27. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  28. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 34
  29. 2019Misleading Authorship Attribution of Source Code using Adversarial Learningcited by 33
  30. 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 33
  31. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 33
  32. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  33. 2022On the Transferability of Pre-trained Language Models for Low-Resource Programming Languagescited by 30
  34. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  35. 2023Large Language Models of Code Fail at Completing Code with Potential Bugscited by 29
  36. 2022CIRCLE: Continual Repair across Programming Languagescited by 17
  37. 2020funcGNN: A Graph Neural Network Approach to Program Similaritycited by 16
  38. 2023From Commit Message Generation to History-Aware Commit Message Completioncited by 16

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 6.

21 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 656
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 230
  3. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
  4. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  5. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  6. 2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 11
  7. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  8. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  9. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  10. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  11. 2024Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Codecited by 3
  12. 2025On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understandingcited by 2
9 more
  1. 2025Can Large Language Models Understand Intermediate Representations in Compilers?cited by 1
  2. 2025Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoderscited by 1
  3. 2026CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corporacited by 1
  4. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 1
  5. 2026Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languagescited by 1
  6. 2024Generating Equivalent Representations of Code By A Self-Reflection Approachcited by 0
  7. 2026LLM Translation of Compiler Intermediate Representationcited by 0
  8. 2026ROSUM-MCTS: Monte Carlo Tree Search-Inspired HDL Code Summarization with Structural Rewardscited by 0
  9. 2026Dependency-Guided Code Generation: Structured Matrix Decomposition and Consistency-Guided Refinementcited by 0