# Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation

arXiv: https://arxiv.org/abs/2407.02056 · HTML: https://mlsanity.com/paper/2407.02056/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Progressive-Hint Prompting Improves Reasoning in Large Language Models](https://mlsanity.com/paper/2304.09797/) (2023)
- [Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://mlsanity.com/paper/2310.03094/) (2023)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives](https://mlsanity.com/paper/2401.02009/) (2024)
- [Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency](https://mlsanity.com/paper/2309.17272/) (2023)
- [BatchEval: Towards Human-like Text Evaluation](https://mlsanity.com/paper/2401.00437/) (2023)
- [Lightweight reranking for language model generations](https://mlsanity.com/paper/2307.06857/) (2023)

## Built on by (most cited first, 21 of 21)

- [Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning](https://mlsanity.com/paper/2408.13457/) (2024)
- [Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling](https://mlsanity.com/paper/2408.17017/) (2024)
- [Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs](https://mlsanity.com/paper/2507.02076/) (2025)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation](https://mlsanity.com/paper/2502.19830/) (2025)
- [Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up](https://mlsanity.com/paper/2410.12323/) (2024)
- [Focused Large Language Models are Stable Many-Shot Learners](https://mlsanity.com/paper/2408.13987/) (2024)
- [Integrative Decoding: Improve Factuality via Implicit Self-consistency](https://mlsanity.com/paper/2410.01556/) (2024)
- [LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient](https://mlsanity.com/paper/2502.01683/) (2025)
- [Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification](https://mlsanity.com/paper/2507.07236/) (2025)
- [Speculative Decoding for Multi-Sample Inference](https://mlsanity.com/paper/2503.05330/) (2025)
- [Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generation](https://mlsanity.com/paper/2503.03106/) (2025)
- [When are likely answers right? On Sequence Probability and Correctness in LLMs](https://mlsanity.com/paper/2606.27359/) (2026)
- [Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Score](https://mlsanity.com/paper/2604.12196/) (2026)
- [Poor-Supervised Evaluation for SuperLLM via Mutual Consistency](https://mlsanity.com/paper/2408.13738/) (2024)
- [Distance Is All You Need: Radial Dispersion for Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2512.04351/) (2025)
- [Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees](https://mlsanity.com/paper/2604.20500/) (2026)
- [Enhancing Hallucination Detection via Future Context](https://mlsanity.com/paper/2507.20546/) (2025)
- [Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs](https://mlsanity.com/paper/2510.03527/) (2025)
- [What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know"](https://mlsanity.com/paper/2604.05779/) (2026)
- [Look Before You Leap: Factual Decoding with Internal Attribution Signals](https://mlsanity.com/paper/2609.15745/) (2026)

## Abstract (arXiv)

Self-consistency (SC), leveraging multiple samples from LLMs, shows significant gains on various reasoning tasks but struggles with free-form generation due to the difficulty of aggregating answers. Its variants, UCS and USC, rely on sample selection or voting mechanisms to improve output quality. These methods, however, face limitations due to their inability to fully utilize the nuanced consensus knowledge present within multiple candidate samples, often resulting in suboptimal outputs. We propose Fine-Grained Self-Consistency (FSC) to addresses these limitations by extracting and integrating segment-level commonalities from candidate samples, enhancing the performance of LLMs both in open-ended and reasoning tasks. Based on this, we present two additional strategies: candidate filtering, which enhances overall quality by identifying highly similar candidate sets, and merging, which reduces input token requirements by combining similar samples. The effectiveness of FSC is demonstrated through extensive experiments on various tasks, including summarization, code generation, and mathematical reasoning, using GPT-3.5-turbo and GPT-4. The results indicate significant improvements over baseline methods, showcasing the potential of FSC to optimize output quality by effectively synthesizing fine-grained consensus knowledge from multiple samples.