mlsanityData to 7 Oct 2026

Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation

arXiv 2407.02056 ↗First posted 2 Jul 2024cs.CL cs.AIcited by 21

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Self-consistency (SC), leveraging multiple samples from LLMs, shows significant gains on various reasoning tasks but struggles with free-form generation due to the difficulty of aggregating answers. Its variants, UCS and USC, rely on sample selection or voting mechanisms to improve output quality. These methods, however, face limitations due to their inability to fully utilize the nuanced consensus knowledge present within multiple candidate samples, often resulting in suboptimal outputs. We propose Fine-Grained Self-Consistency (FSC) to addresses these limitations by extracting and integrating segment-level commonalities from candidate samples, enhancing the performance of LLMs both in open-ended and reasoning tasks. Based on this, we present two additional strategies: candidate filtering, which enhances overall quality by identifying highly similar candidate sets, and merging, which reduces input token requirements by combining similar samples. The effectiveness of FSC is demonstrated through extensive experiments on various tasks, including summarization, code generation, and mathematical reasoning, using GPT-3.5-turbo and GPT-4. The results indicate significant improvements over baseline methods, showcasing the potential of FSC to optimize output quality by effectively synthesizing fine-grained consensus knowledge from multiple samples.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 18 of its 29 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

18 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives4 Jan
  2. 2023Universal Self-Consistency for Large Language Model Generation29 Nov

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2023Large Language Models as Optimizerscited by 830
  9. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  10. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  11. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  12. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
4 more
  1. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  2. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  3. 2023BatchEval: Towards Human-like Text Evaluationcited by 16
  4. 2023Lightweight reranking for language model generationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  2. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoningcited by 61
  3. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  4. 2024A Survey on the Honesty of Large Language Modelscited by 21
  5. 2025Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregationcited by 11
  6. 2024Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-upcited by 10
  7. 2024Focused Large Language Models are Stable Many-Shot Learnerscited by 8
  8. 2024Integrative Decoding: Improve Factuality via Implicit Self-consistencycited by 8
  9. 2025LLM-Powered Benchmark Factory: Reliable, Generic, and Efficientcited by 6
  10. 2025Speculative Decoding for Multi-Sample Inferencecited by 6
  11. 2025Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantificationcited by 6
  12. 2025Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generationcited by 4
9 more
  1. 2026When are likely answers right? On Sequence Probability and Correctness in LLMscited by 3
  2. 2026Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Scorecited by 2
  3. 2024Poor-Supervised Evaluation for SuperLLM via Mutual Consistencycited by 1
  4. 2025Distance Is All You Need: Radial Dispersion for Uncertainty Estimation in Large Language Modelscited by 1
  5. 2026Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Treescited by 1
  6. 2025Enhancing Hallucination Detection via Future Contextcited by 0
  7. 2025Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrscited by 0
  8. 2026What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know"cited by 0
  9. 2026Look Before You Leap: Factual Decoding with Internal Attribution Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
  2. 2026Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning
  3. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Models
  4. 2025Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
  5. 2025Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs
  6. 2024Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting
  7. 2026Agreement in Representation Space for Open-Ended Self-Consistency
  8. 2024Improving Self Consistency in LLMs through Probabilistic Tokenization