# In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search

arXiv: https://arxiv.org/abs/2311.07237 · HTML: https://mlsanity.com/paper/2311.07237/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Explore, Establish, Exploit: Red Teaming Language Models from Scratch](https://mlsanity.com/paper/2306.09442/) (2023)
- [Adversarial Examples: Opportunities and Challenges](https://mlsanity.com/paper/1809.04790/) (2018)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge](https://mlsanity.com/paper/2305.05976/) (2023)
- [RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms](https://mlsanity.com/paper/2005.00782/) (2020)
- [Open Knowledge Enrichment for Long-tail Entities](https://mlsanity.com/paper/2002.06397/) (2020)
- [Benchmarking Long-tail Generalization with Likelihood Splits](https://mlsanity.com/paper/2210.06799/) (2022)
- [UnCommonSense: Informative Negative Knowledge about Everyday Concepts](https://mlsanity.com/paper/2208.09292/) (2022)
- [Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses](https://mlsanity.com/paper/2305.19339/) (2023)

## Built on by (most cited first, 9 of 9)

- [Empowering LLMs with Logical Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2502.15652/) (2025)
- [AI and the Problem of Knowledge Collapse](https://mlsanity.com/paper/2404.03502/) (2024)
- [GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation](https://mlsanity.com/paper/2505.20416/) (2025)
- [Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory](https://mlsanity.com/paper/2605.31086/) (2026)
- [Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization](https://mlsanity.com/paper/2405.02517/) (2024)
- [Taming Data Challenges in ML-based Security Tasks Using Generative AI](https://mlsanity.com/paper/2507.06092/) (2025)
- [TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models](https://mlsanity.com/paper/2511.21006/) (2025)
- [TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entities](https://mlsanity.com/paper/2603.27768/) (2026)
- [RAGas: Retrieval-Augmented Gas Optimization for Smart Contracts with Continuous Knowledge Integration](https://mlsanity.com/paper/2608.15857/) (2026)

## Abstract (arXiv)

To effectively use large language models (LLMs) for real-world queries, it is imperative that they generalize to the long-tail distribution, i.e. rare examples where models exhibit low confidence. In this work, we take the first step towards evaluating LLMs in the long-tail distribution of inferential knowledge. We exemplify long-tail evaluation on the Natural Language Inference task. First, we introduce Logic-Induced-Knowledge-Search (LINK), a systematic long-tail data generation framework, to obtain factually-correct yet long-tail inferential statements. LINK uses variable-wise prompting grounded on symbolic rules to seek low-confidence statements while ensuring factual correctness. We then use LINK to curate Logic-Induced-Long-Tail (LINT), a large-scale long-tail inferential knowledge dataset that contains 108K statements spanning four domains. We evaluate popular LLMs on LINT; we find that state-of-the-art LLMs show significant performance drop (21% relative drop for GPT4) on long-tail data as compared to on head distribution data, and smaller models show even more generalization weakness. These results further underscore the necessity of long-tail evaluation in developing generalizable LLMs.