# On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Software

arXiv: https://arxiv.org/abs/2311.11979 · HTML: https://mlsanity.com/paper/2311.11979/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Metamorphic Testing: A New Approach for Generating Next Test Cases](https://mlsanity.com/paper/2002.12543/) (2020)
- [A Survey on Semantic Parsing](https://mlsanity.com/paper/1812.00978/) (2018)
- [Few-Shot Semantic Parsing with Language Models Trained On Code](https://mlsanity.com/paper/2112.08696/) (2021)
- [Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances](https://mlsanity.com/paper/2204.10825/) (2022)
- [Semantic Parsing with Syntax- and Table-Aware SQL Generation](https://mlsanity.com/paper/1804.08338/) (2018)
- [Assisting Language Learners: Automated Trans-Lingual Definition Generation via Contrastive Prompt Learning](https://mlsanity.com/paper/2306.06058/) (2023)
- [Metamorphic Testing and Debugging of Tax Preparation Software](https://mlsanity.com/paper/2205.04998/) (2022)

## Built on by (most cited first, 3 of 3)

- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- [An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software](https://mlsanity.com/paper/2509.13471/) (2025)

## Abstract (arXiv)

Due to the ever-increasing complexity of income tax laws in the United States, the number of US taxpayers filing their taxes using tax preparation software (henceforth, tax software) continues to increase. According to the U.S. Internal Revenue Service (IRS), in FY22, nearly 50% of taxpayers filed their individual income taxes using tax software. Given the legal consequences of incorrectly filing taxes for the taxpayer, ensuring the correctness of tax software is of paramount importance. Metamorphic testing has emerged as a leading solution to test and debug legal-critical tax software due to the absence of correctness requirements and trustworthy datasets. The key idea behind metamorphic testing is to express the properties of a system in terms of the relationship between one input and its slightly metamorphosed twinned input. Extracting metamorphic properties from IRS tax publications is a tedious and time-consuming process. As a response, this paper formulates the task of generating metamorphic specifications as a translation task between properties extracted from tax documents - expressed in natural language - to a contrastive first-order logic form. We perform a systematic analysis on the potential and limitations of in-context learning with Large Language Models(LLMs) for this task, and outline a research agenda towards automating the generation of metamorphic specifications for tax preparation software.