mlsanityData to 7 Oct 2026

Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation

arXiv 2401.10186 ↗First posted 18 Jan 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design Quintd - a tool for collecting novel structured data records from public APIs. We find that open LLMs (Llama 2, Mistral, and Zephyr) can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. However, we show that the semantic accuracy of the outputs is a major issue: both according to human annotators and our reference-free metric based on GPT-4, more than 80% of the outputs of open LLMs contain at least one semantic error. We publicly release the code, data, and model outputs.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 46 of its 88 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

46 of 88 references.

References · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
34 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  2. 2020Evaluation of Text Generation: A Surveycited by 275
  3. 2023How is ChatGPT's behavior changing over time?cited by 265
  4. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  5. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  6. 2017Challenges in Data-to-Document Generationcited by 226
  7. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  8. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  9. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  10. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  11. 2019Text Generation from Knowledge Graphs with Graph Transformerscited by 122
  12. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  13. 2018Data-to-Text Generation with Content Selection and Planningcited by 106
  14. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  15. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  16. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
  17. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  18. 2019Data-to-text Generation with Entity Modelingcited by 45
  19. 2019Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialoguecited by 35
  20. 2020A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systemscited by 34
  21. 2021Data-to-text Generation with Macro Planningcited by 34
  22. 2020Revisiting Challenges in Data-to-Text Generation with Fact Groundingcited by 27
  23. 2023Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphscited by 24
  24. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 21
  25. 2019A Hierarchical Model for Data-to-Text Generationcited by 20
  26. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  27. 2016Multi-domain Neural Network Language Generation for Spoken Dialogue Systemscited by 18
  28. 2022Data-to-text Generation with Variational Sequential Planningcited by 17
  29. 2023Evaluating Generative Models for Graph-to-Text Generationcited by 16
  30. 2021Controllable and Diverse Text Generation in E-commercecited by 13
  31. 2023Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarioscited by 12
  32. 2022Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Modelscited by 6
  33. 2019WikiDataSets: Standardized sub-graphs from Wikidatacited by 5
  34. 2023Data-to-text Generation for Severely Under-Resourced Languages with GPT-3.5: A Bit of Help Needed from Google Translatecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  3. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluationcited by 7
  4. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  5. 2025MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generationcited by 6
  6. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  7. 2025StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluationcited by 4
  8. 2025The PacifAIst Benchmark:Would an Artificial Intelligence Choose to Sacrifice Itself for Human Safety?cited by 4
  9. 2025SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMscited by 3
  10. 2024factgenie: A Framework for Span-based Evaluation of Generated Textscited by 2
  11. 2025WebNovelBench: Placing LLM Novelists on the Web Novel Distributioncited by 2
  12. 2026Strategies for Span Labeling with Large Language Modelscited by 2
4 more
  1. 2025Representing data in words: A context engineering approachcited by 1
  2. 2025FreshTab: Sourcing Fresh Data for Table-to-Text Generation Evaluationcited by 1
  3. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0
  4. 2026Do LLMs Make More Mistakes If They Do Not Believe the Input Data?cited by 0