mlsanityData to 7 Oct 2026

StructuredRAG: JSON Response Formatting with Large Language Models

arXiv 2408.11061 ↗First posted 7 Aug 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON, is crucial for their use in Compound AI Systems. However, evaluating and improving this capability remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions. We evaluate two state-of-the-art LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting. Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro. We observe that task complexity significantly influences performance, with tasks involving lists or composite object outputs proving more challenging. Our findings highlight the need for further research into improving the reliability and consistency of structured output generation in LLMs. We have open-sourced our experimental code and results at github.com/weaviate/structured-rag.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

26 of 30 references.

References · 26

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2023Large Language Models as Optimizerscited by 830
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  12. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
14 more
  1. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  2. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  3. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  4. 2024More Agents Is All You Needcited by 174
  5. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  6. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 99
  7. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 90
  8. 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
  9. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 38
  10. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  11. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  12. 2024Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Designcited by 13
  13. 2024Alto: Orchestrating Distributed Compound AI Systems with Nested Ancestrycited by 7
  14. 2024Guiding Large Language Models to Generate Computer-Parsable Contentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 7.

22 papers.

Sort:

By area:

  1. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
  2. 2025Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherencecited by 16
  3. 2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14
  4. 2025REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Oncecited by 12
  5. 2025Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Modelscited by 9
  6. 2025Evaluating open-source Large Language Models for automated fact-checkingcited by 6
  7. 2025End-to-end PDDL Planning with Hardcoded and Dynamic Agentscited by 5
  8. 2025STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliabilitycited by 5
  9. 2025LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMscited by 3
  10. 2025Querying Databases with Function Callingcited by 2
  11. 2025DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correctioncited by 2
  12. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answeringcited by 2
10 more
  1. 2026ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflowscited by 2
  2. 2025How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performancecited by 1
  3. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  4. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehensioncited by 1
  5. 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
  6. 2025Synthetic Homes: A Multimodal Generative AI Pipeline for Residential Building Data Generation under Data Scarcitycited by 0
  7. 2025Orchestrator Multi-Agent Clinical Decision Support System for Secondary Headache Diagnosis in Primary Carecited by 0
  8. 2026Generating Hierarchical JSON Representations of Scientific Sentences Using LLMscited by 0
  9. 2026Implicit Framing in Obstetric Counseling Notes: A Grounded LLM Pipeline on a VBAC-Eligible Cohortcited by 0
  10. 2026Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languagescited by 0