mlsanityData to 7 Oct 2026

Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL

arXiv 2205.12422 ↗First posted 25 May 2022cs.CL cs.AI cs.PLcited by 12

Not on the map. Why?

Abstract (arXiv)

Can non-programmers annotate natural language utterances with complex programs that represent their meaning? We introduce APEL, a framework in which non-programmers select among candidate programs generated by a seed semantic parser (e.g., Codex). Since they cannot understand the candidate programs, we ask them to select indirectly by examining the programs' input-ouput examples. For each utterance, APEL actively searches for a simple input on which the candidate programs tend to produce different outputs. It then asks the non-programmers only to choose the appropriate output, thus allowing us to infer which program is correct and could be used to fine-tune the parser. As a first case study, we recruited human non-programmers to use APEL to re-annotate SPIDER, a text-to-SQL dataset. Our approach achieved the same annotation accuracy as the original expert annotators (75%) and exposed many subtle errors in the original annotations.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 27 of its 60 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

27 of 60 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  9. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  11. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  12. 2022Predictability and Surprise in Large Generative Modelscited by 209
15 more
  1. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  2. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  3. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  4. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  5. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  6. 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
  7. 2021Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLPcited by 30
  8. 2020TF-Coder: Program Synthesis for Tensor Manipulationscited by 27
  9. 2020An Imitation Game for Learning Semantic Parsers from User Interactioncited by 23
  10. 2020DuoRAT: Towards Simpler Text-to-SQL Modelscited by 17
  11. 2020Benchmarking Multimodal Regex Synthesis with Complex Structurescited by 10
  12. 2021Towards Transparent Interactive Semantic Parsing via Step-by-Step Correctioncited by 9
  13. 2021Falx: Synthesis-Powered Visualization Authoringcited by 8
  14. 2016Inferring Logical Forms From Denotationscited by 6
  15. 2019Explaining Wrong Queries Using Small Examplescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 5; 2025: 3; 2026: 2.

12 papers.

  1. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  2. 2022Coder Reviewer Reranking for Code Generationcited by 83
  3. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  4. 2024A Survey of Table Reasoning with Large Language Modelscited by 29
  5. 2022HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Modelscited by 17
  6. 2025MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Toolscited by 15
  7. 2025SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQLcited by 15
  8. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11
  9. 2024DAC: Decomposed Automation Correction for Text-to-SQLcited by 6
  10. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trustcited by 1
  11. 2025COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontiercited by 0
  12. 2026Harnessing the Latent Space: From Steering Vectors to Model Calibrators for Control and Trustcited by 0