mlsanityData to 7 Oct 2026

Leveraging Code to Improve In-context Learning for Semantic Parsing

arXiv 2311.09519 ↗First posted 16 Nov 2023cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. However, learning to parse to rare domain-specific languages (DSLs) from just a few demonstrations is challenging, limiting the performance of even the most capable LLMs. In this work, we improve the effectiveness of ICL for semantic parsing by (1) using general-purpose programming languages such as Python instead of DSLs, and (2) augmenting prompts with a structured domain description that includes, e.g., the available classes and functions. We show that both these changes significantly improve accuracy across three popular datasets. Combined, they lead to dramatic improvements (e.g. 7.9% to 66.5% on SMCalFlow compositional split), nearly closing the performance gap between easier i.i.d. and harder compositional splits when used with a strong model, and reducing the need for a large number of demonstrations. We find that the resemblance of the target parse language to general-purpose code is a more important factor than the language's popularity in pre-training corpora. Our findings provide an improved methodology for building semantic parsers in the modern context of ICL with LLMs.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 37 of its 75 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

37 of 75 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  6. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  7. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  8. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  9. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  10. 2022Active Example Selection for In-Context Learningcited by 195
  11. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  12. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
25 more
  1. 2019Good-Enough Compositional Data Augmentationcited by 158
  2. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  3. 2023Compositional Exemplars for In-context Learningcited by 143
  4. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  5. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  6. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  7. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  8. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  9. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  10. 2023Coverage-based Example Selection for In-Context Learningcited by 58
  11. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  12. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  13. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  14. 2020AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Datacited by 29
  15. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  16. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
  17. 2022Structural generalization is hard for sequence-to-sequence modelscited by 24
  18. 2022On the Compositional Generalization Gap of In-Context Learningcited by 24
  19. 2023ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretationcited by 24
  20. 2020Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answeringcited by 16
  21. 2018Decoupling Structure and Lexicon for Zero-Shot Semantic Parsingcited by 11
  22. 2023Natural Language Decomposition and Interpretation of Complex Utterancescited by 10
  23. 2022Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluationcited by 8
  24. 2023Exploring the Curious Case of Code Promptscited by 6
  25. 2023Compositional Generalization without Trees using Multiset Tagging and Latent Permutationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 1.

10 papers.

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024ReGAL: Refactoring Programs to Discover Generalizable Abstractionscited by 32
  3. 2024Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimizationcited by 29
  4. 2024TACT: Advancing Complex Aggregative Reasoning with Information Extraction Toolscited by 5
  5. 2025Read it in Two Steps: Translating Extremely Low-Resource Languages with Code-Augmented Grammar Bookscited by 4
  6. 2025Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learningcited by 1
  7. 2025Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsingcited by 1
  8. 2025ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Executioncited by 1
  9. 2025Natural Language Interaction with Databases on Edge Devices in the Internet of Battlefield Thingscited by 0
  10. 2026Accelerating Constrained Decoding with Token Space Compressioncited by 0