# Investigating Answerability of LLMs for Long-Form Question Answering

arXiv: https://arxiv.org/abs/2309.08210 · HTML: https://mlsanity.com/paper/2309.08210/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)

## Built on by (most cited first, 9 of 9)

- [Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices](https://mlsanity.com/paper/2403.12503/) (2024)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text](https://mlsanity.com/paper/2403.05750/) (2024)
- [Multi-Document Financial Question Answering using LLMs](https://mlsanity.com/paper/2411.07264/) (2024)
- [Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter](https://mlsanity.com/paper/2408.10573/) (2024)
- [A Benchmark Dataset with Larger Context for Non-Factoid Question Answering over Islamic Text](https://mlsanity.com/paper/2409.09844/) (2024)
- [Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora](https://mlsanity.com/paper/2505.08905/) (2025)
- [Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.20825/) (2025)
- [AIVA: An AI-based Virtual Companion for Emotion-aware Interaction](https://mlsanity.com/paper/2509.03212/) (2025)

## Abstract (arXiv)

As we embark on a new era of LLMs, it becomes increasingly crucial to understand their capabilities, limitations, and differences. Toward making further progress in this direction, we strive to build a deeper understanding of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective open-source LLMs and their distilled counterparts. To this end, we specifically focus on long-form question answering (LFQA) because it has several practical and impactful applications (e.g., troubleshooting, customer service, etc.) yet is still understudied and challenging for LLMs. We propose a question-generation method from abstractive summaries and show that generating follow-up questions from summaries of long documents can create a challenging setting for LLMs to reason and infer from long contexts. Our experimental results confirm that: (1) our proposed method of generating questions from abstractive summaries pose a challenging setup for LLMs and shows performance gaps between LLMs like ChatGPT and open-source LLMs (Alpaca, Llama) (2) open-source LLMs exhibit decreased reliance on context for generated questions from the original document, but their generation capabilities drop significantly on generated questions from summaries -- especially for longer contexts (>1024 tokens)