mlsanityData to 14 Oct 2026

Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected values

arXiv 2406.10267 ↗First posted 11 Jun 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

LLM text decoding is key component for perceived LLM quality. We demonstrate two experiments showing that decoding methods could be improved by manipulation of token probabilities. First, we test few LLM on SummEval summary scoring dataset, to measure reading comprehension. We compare scores from greedy decoding to expected values over the next token distribution. We scale logits by large temperature to increase the entropy of scores. This allows strong improvement of performance on SummEval (in terms of correlations to human judgement). We see improvement from 6-8% to 13-28% for 7B Mistral and from 20%-46% to 37%-56% for Mixtral, beating GPT 4 0314 result on two metrics. Part of the gain seems related to positional bias. Secondly, we use probability-based tree sampling algorithm, to examine all most probable generations for given prompt.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

18 of 26 references.

References · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Mistral 7Bcited by 3,164
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024Mixtral of Expertscited by 1,823
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  10. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  11. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  12. 2022Locally Typical Samplingcited by 111
6 more
  1. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  2. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  3. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 36
  4. 2022Arithmetic Sampling: Parallel Diverse Decoding for Large Language Modelscited by 19
  5. 2021Determinantal Beam Searchcited by 18
  6. 2024Priority Sampling of Large Language Models for Compilerscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  2. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  3. 2025Explainability-Based Token Replacement on LLM-Generated Textcited by 3
  4. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1