# FLAME: A small language model for spreadsheet formulas

arXiv: https://arxiv.org/abs/2301.13779 · HTML: https://mlsanity.com/paper/2301.13779/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 80 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 30 of 35)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [Graph-based, Self-Supervised Program Repair from Diagnostic Feedback](https://mlsanity.com/paper/2005.10636/) (2020)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)

## Built on by (most cited first, 15 of 15)

- [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023)
- [SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models](https://mlsanity.com/paper/2305.19308/) (2023)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [SpreadsheetLLM: Encoding Spreadsheets for Large Language Models](https://mlsanity.com/paper/2407.09025/) (2024)
- [SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows](https://mlsanity.com/paper/2606.29955/) (2026)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [TableTalk: Scaffolding Spreadsheet Development with a Language Agent](https://mlsanity.com/paper/2502.09787/) (2025)
- [Knowledge Base Question Answering for Space Debris Queries](https://mlsanity.com/paper/2305.19734/) (2023)
- [Formula-R1: Incentivizing LLM Reasoning over Complex Tables with Numerical Computation via Formula-Driven Reinforcement Learning](https://mlsanity.com/paper/2505.23667/) (2025)
- [Diffusion is a code repair operator and generator](https://mlsanity.com/paper/2508.11110/) (2025)
- [Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)](https://mlsanity.com/paper/2501.05465/) (2025)
- [SODBench: A Large Language Model Approach to Documenting Spreadsheet Operations](https://mlsanity.com/paper/2510.19864/) (2025)
- [AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMs](https://mlsanity.com/paper/2512.15764/) (2025)
- [An Empirical Study of Validating Synthetic Data for Formula Generation](https://mlsanity.com/paper/2407.10657/) (2024)
- [Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs](https://mlsanity.com/paper/2508.11715/) (2025)

## Abstract (arXiv)

Spreadsheets are a vital tool for end-user data management. Using large language models for formula authoring assistance in these environments can be difficult, as these models are expensive to train and challenging to deploy due to their size (up to billions of parameters). We present FLAME, a transformer-based model trained exclusively on Excel formulas that leverages domain insights to achieve competitive performance while being substantially smaller (60M parameters) and training on two orders of magnitude less data. We curate a training dataset using sketch deduplication, introduce an Excel-specific formula tokenizer, and use domain-specific versions of masked span prediction and noisy auto-encoding as pre-training objectives. We evaluate FLAME on formula repair, formula completion, and similarity-based formula retrieval. FLAME can outperform much larger models, such as the Davinci (175B) and Cushman (12B) variants of Codex and CodeT5 (220M), in 10 of 14 evaluation settings for the repair and completion tasks. For formula retrieval, FLAME outperforms CodeT5, CodeBERT, and GraphCodeBERT.