# Automated Alignment between Elicitation Interviews and Requirements

arXiv: https://arxiv.org/abs/2510.08622 · HTML: https://mlsanity.com/paper/2510.08622/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Automated Fact-Checking for Assisting Human Fact-Checkers](https://mlsanity.com/paper/2103.07769/) (2021)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?](https://mlsanity.com/paper/2208.09214/) (2022)
- [ChatGPT as a tool for User Story Quality Evaluation: Trustworthy Out of the Box?](https://mlsanity.com/paper/2306.12132/) (2023)
- [Requirements Engineering using Generative AI: Prompts and Prompting Patterns](https://mlsanity.com/paper/2311.03832/) (2023)
- [LLMREI: Automating Requirements Elicitation Interviews with LLMs](https://mlsanity.com/paper/2507.02564/) (2025)
- [Naming the Pain in Requirements Engineering: Contemporary Problems, Causes, and Effects in Practice](https://mlsanity.com/paper/1611.10288/) (2016)
- [Leveraging LLMs for User Stories in AI Systems: UStAI Dataset](https://mlsanity.com/paper/2504.00513/) (2025)
- [Can LLMs Generate User Stories and Assess Their Quality?](https://mlsanity.com/paper/2507.15157/) (2025)
- [Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys](https://mlsanity.com/paper/1805.07951/) (2018)

## Abstract (arXiv)

Software requirements are derived from a variety of elicitation techniques, many of which have a conversational nature, like interviews. However, evaluating whether those derived requirements faithfully reflect the stakeholders' needs remains a challenging manual task. In this paper, we formalize the task of aligning the transcript of an interview with a collection of requirements represented as user stories. We propose two heuristic metrics for alignment, called (i) requirements faithfulness: the proportion of stories supported by the transcript, and (ii) interview coverage: the proportion of transcript supported by at least one story. Then, we run experiments with large language models and embedding models that assess the ability of evaluating these metrics automatically. Experiments over four datasets show that an LLM-based solution achieves 0.86 macro-F1 on manually labeled chunk-story pairs. We also show how embedding models can be used as blockers to make the approach more scalable. This work paves the way for more research on linking conversational artifacts with requirements. The formal framework and the automated matching techniques are basic components that can be used for emerging tasks such as tracing requirements to interviews and generating requirements from conversations.