mlsanityData to 7 Oct 2026

Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?

arXiv 2410.15512 ↗First posted 20 Oct 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Question answering (QA), giving correct answers to questions, is a popular task, but we test reverse question answering (RQA): for an input answer, give a question with that answer. Past work tests QA and RQA separately, but we test them jointly, comparing their difficulty, aiding benchmark design, and checking reasoning consistency. We run 16 LLMs on QA and RQA with trivia questions/answers, revealing: 1) Versus QA, LLMs are much less accurate in RQA for numerical answers, but slightly more accurate in RQA for textual answers; 2) LLMs often answer their own invalid questions from RQA accurately in QA, so RQA errors are not from knowledge gaps alone; 3) RQA errors correlate with question difficulty and inversely correlate with answer frequencies in the Dolma corpus; and 4) LLMs struggle to provide valid multi-hop questions. By finding question and answer types that lead to RQA errors, we suggest improvements for LLM reasoning.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

39 of 65 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  5. 2024Yi: Open Foundation Models by 01.AIcited by 706
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  7. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
27 more
  1. 2023How Language Model Hallucinations Can Snowballcited by 310
  2. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  4. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  7. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  8. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  9. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  10. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 70
  11. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  12. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  13. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  14. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  15. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  16. 2019Quizbowl: The Case for Incremental Question Answeringcited by 36
  17. 2022Language Models as Inductive Reasonerscited by 32
  18. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  19. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Languagecited by 29
  20. 2021Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrievalcited by 25
  21. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 24
  22. 2021Question Generation for Adaptive Educationcited by 22
  23. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  24. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoningcited by 13
  25. 2023It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoningcited by 12
  26. 2023Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanationscited by 11
  27. 2020A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answeringcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  2. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  3. 2025Theorem Prover as a Judge for Synthetic Data Generationcited by 13
  4. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  5. 2025QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generationcited by 10
  6. 2025What Has Been Lost with Synthetic Evaluation?cited by 8
  7. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  8. 2025EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videoscited by 3
  9. 2026BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarkscited by 3
  10. 2025Principled Content Selection to Generate Diverse and Personalized Multi-Document Summariescited by 1
  11. 2025HANRAG: Heuristic Accurate Noise-resistant Retrieval-Augmented Generation for Multi-hop Question Answeringcited by 1
  12. 2025Multi-granular Training Strategies for Robust Multi-hop Reasoning Over Noisy and Heterogeneous Knowledge Sourcescited by 0
1 more
  1. 2026Time Capsule of Testable Human Knowledge: 41 Years of Jeopardy! in a Single Free Local Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
  2. 2026Inferential Question Answering
  3. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Survey
  4. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
  5. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth
  6. 2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs
  7. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models
  8. 2025Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong