mlsanityData to 14 Oct 2026

Lyra: A Benchmark for Turducken-Style Code Generation

arXiv 2108.12144 ↗First posted 27 Aug 2021cs.SE cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Recently, neural techniques have been used to generate source code automatically. While promising for declarative languages, these approaches achieve much poorer performance on datasets for imperative languages. Since a declarative language is typically embedded in an imperative language (i.e., the turducken-style programming) in real-world software development, the promising results on declarative languages can hardly lead to significant reduction of manual software development efforts. In this paper, we define a new code generation task: given a natural language comment, this task aims to generate a program in a base imperative language with an embedded declarative language. To our knowledge, this is the first turducken-style code generation task. For this task, we present Lyra: a dataset in Python with embedded SQL. This dataset contains 2,000 carefully annotated database manipulation programs from real-world projects. Each program is paired with both a Chinese comment and an English comment. In our experiment, we adopted Transformer, BERT-style, and GPT-style models as baselines. In the best setting, the generation performance of GPT-style models is better than others, where the AST exact matching accuracy is 24% and 25.5% when using Chinese and English comments, respectively. Therefore, we believe that Lyra provides a new challenge for code generation. Yet, overcoming this challenge may significantly boost the applicability of code generation techniques for real-world software development.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 23 of its 45 references that are in the map. First posted on arXiv 27 Aug 2021.

Builds on

23 of 45 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  9. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  10. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  11. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  12. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
11 more
  1. 2018Mapping Language to Code in Programmatic Contextcited by 104
  2. 2017Learning a Neural Semantic Parser from User Feedbackcited by 99
  3. 2020SmBoP: Semi-autoregressive Bottom-up Semantic Parsingcited by 97
  4. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  5. 2016Latent Predictor Networks for Code Generationcited by 73
  6. 2019TreeGen: A Tree-Based Transformer Architecture for Code Generationcited by 58
  7. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  8. 2018Retrieval-Based Neural Code Generationcited by 41
  9. 2021SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoisingcited by 27
  10. 2021Improving Text-to-SQL with Schema Dependency Learningcited by 25
  11. 2020TransRegex: Multi-modal Regular Expression Synthesis by Generate-and-Repaircited by 8

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1; 2025: 1.

5 papers.

  1. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  2. 2023CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generationcited by 8
  3. 2024Defending Code Language Models against Backdoor Attacks with Deceptive Cross-Entropy Losscited by 5
  4. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  5. 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3