mlsanityData to 7 Oct 2026

InstructExcel: A Benchmark for Natural Language Instruction in Excel

arXiv 2310.14495 ↗First posted 23 Oct 2023cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets. This work investigates whether LLMs can generate code (Excel OfficeScripts, a TypeScript API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale benchmark, InstructExcel, created by leveraging the 'Automate' feature in Excel to automatically generate OfficeScripts from users' actions. Our benchmark includes over 10k samples covering 170+ Excel operations across 2,000 publicly available Excel spreadsheets. Experiments across various zero-shot and few-shot settings show that InstructExcel is a hard benchmark for state of the art models like GPT-4. We observe that (1) using GPT-4 over GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can help improve performance on this benchmark.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

44 of 54 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
32 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  4. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  5. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  6. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  7. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  8. 2022Can Foundation Models Wrangle Your Data?cited by 173
  9. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  10. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  11. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  12. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  13. 2020Learning from Task Descriptionscited by 78
  14. 2018A Survey on Semantic Parsingcited by 62
  15. 2021Few-Shot Semantic Parsing with Language Models Trained On Codecited by 50
  16. 2015Program Synthesis using Natural Languagecited by 41
  17. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41
  18. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  19. 2022Is a Question Decomposition Unit All We Need?cited by 37
  20. 2022InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NERcited by 31
  21. 2023Instruction Tuned Models are Quick Learnerscited by 21
  22. 2022Less is More: Summary of Long Instructions is Better for Program Synthesiscited by 17
  23. 2022HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Modelscited by 17
  24. 2022How Many Data Samples is an Additional Instruction Worth?cited by 16
  25. 2022Pop Quiz! Can a Large Language Model Help With Reverse Engineering?cited by 13
  26. 2022BioTABQA: Instruction Learning for Biomedical Table Question Answeringcited by 13
  27. 2021Lux: Always-on Visualization Recommendations for Exploratory Dataframe Workflowscited by 10
  28. 2022Compositional Task-Oriented Parsing as Abstractive Question Answeringcited by 10
  29. 2022On the Effectiveness of Pretrained Models for API Learningcited by 9
  30. 2022Compositional Generalization and Decomposition in Neural Program Synthesiscited by 6
  31. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  32. 2022CORNET: Learning Table Formatting Rules By Examplecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 100
  2. 2024SheetAgent: Towards A Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Modelscited by 16
  3. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  4. 2026SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflowscited by 7
  5. 2025TableTalk: Scaffolding Spreadsheet Development with a Language Agentcited by 5
  6. 2026From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understandingcited by 5
  7. 2025Diffusion is a code repair operator and generatorcited by 3
  8. 2026Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editingcited by 3
  9. 2024WikiTableEdit: A Benchmark for Table Editing by Natural Language Instructioncited by 2
  10. 2026DV-World: Benchmarking Data Visualization Agents in Real-World Scenarioscited by 1
  11. 2025Synthetic Function Demonstrations Improve Generation in Low-Resource Programming Languagescited by 0
  12. 2025Tabularis Formatus: Predictive Formatting for Tablescited by 0
2 more
  1. 2026CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMscited by 0
  2. 2026Neurosymbolics for Data Engineering: Achieving Long Context Token Reduction Without Finetuningcited by 0