# Learning to Generate Structured Output with Schema Reinforcement Learning

arXiv: https://arxiv.org/abs/2502.18878 · HTML: https://mlsanity.com/paper/2502.18878/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](https://mlsanity.com/paper/2305.13971/) (2023)
- [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)
- [WebCPM: Interactive Web Search for Chinese Long-form Question Answering](https://mlsanity.com/paper/2305.06849/) (2023)
- [FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability](https://mlsanity.com/paper/2402.18667/) (2024)
- [JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models](https://mlsanity.com/paper/2501.10868/) (2025)
- [Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model](https://mlsanity.com/paper/2310.05155/) (2023)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [Verifiable Format Control for Large Language Model Generations](https://mlsanity.com/paper/2502.04498/) (2025)
- [LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models](https://mlsanity.com/paper/2502.02441/) (2025)

## Built on by (most cited first, 29 of 29)

- [A Technical Survey of Reinforcement Learning Techniques for Large Language Models](https://mlsanity.com/paper/2507.04136/) (2025)
- [SLOT: Structuring the Output of Large Language Models](https://mlsanity.com/paper/2505.04016/) (2025)
- [Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis](https://mlsanity.com/paper/2602.00846/) (2026)
- [PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction](https://mlsanity.com/paper/2510.08623/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [LLMStructBench: Benchmarking Large Language Model Structured Data Extraction](https://mlsanity.com/paper/2602.14743/) (2026)
- [When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output](https://mlsanity.com/paper/2503.24191/) (2025)
- [Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciences](https://mlsanity.com/paper/2507.21831/) (2025)
- [Training-Free Multimodal Large Language Model Orchestration](https://mlsanity.com/paper/2508.10016/) (2025)
- [The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs](https://mlsanity.com/paper/2605.08737/) (2026)
- [Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety](https://mlsanity.com/paper/2605.12729/) (2026)
- [MeXtract: Light-Weight Metadata Extraction from Scientific Papers](https://mlsanity.com/paper/2510.06889/) (2025)
- [ADEMA: A Knowledge-State Orchestration Architecture for Long-Horizon Knowledge Synthesis with LLMAgents](https://mlsanity.com/paper/2604.25849/) (2026)
- [CuraView: A Multi-Agent Framework for Medical Hallucination Detection with GraphRAG-Enhanced Knowledge Verification](https://mlsanity.com/paper/2605.03476/) (2026)
- [Structured Document Translation via Format Reinforcement Learning](https://mlsanity.com/paper/2512.05100/) (2025)
- [CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs](https://mlsanity.com/paper/2512.19305/) (2025)
- [ALGOGEN: Tool-Generated Verifiable Traces for Reliable Algorithm Visualization](https://mlsanity.com/paper/2605.12159/) (2026)
- [SPIN: Structural LLM Planning via Iterative Navigation for Industrial Tasks](https://mlsanity.com/paper/2605.14051/) (2026)
- [Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature](https://mlsanity.com/paper/2606.05206/) (2026)
- [STAGE: Source-grounded Text-to-JSON Artifact GEneration](https://mlsanity.com/paper/2606.20072/) (2026)
- [ASMR: Agentic Schema Generation for Ship Maintenance Report Writing](https://mlsanity.com/paper/2607.08177/) (2026)
- [QuantumMind: Constraint-Grounded Agentic Reasoning for Speedup Analysis in Quantum Computing](https://mlsanity.com/paper/2608.07743/) (2026)
- [The Parser Already Knows: Lightweight Bias Correction in Constrained Decoding](https://mlsanity.com/paper/2608.10137/) (2026)
- [TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model](https://mlsanity.com/paper/2608.15223/) (2026)
- [Grounded Normative Rule Generation with Structured Search](https://mlsanity.com/paper/2608.22229/) (2026)
- [GrammarRL: Effective Grammar-Constrained Decoding via Reinforcement Learning](https://mlsanity.com/paper/2609.39869/) (2026)
- [ContractRL: Shielded Group-Relative Policy Optimization for Auditable Tool-Call Repair](https://mlsanity.com/paper/2610.00328/) (2026)
- [Structural Chain-of-Thought: Reasoning with Executable Computational Graphs](https://mlsanity.com/paper/2610.15107/) (2026)
- [ReFormBench: Benchmarking the Stability of Response-Conditioned Format Correction in Large Language Models](https://mlsanity.com/paper/2610.19693/) (2026)

## Abstract (arXiv)

This study investigates the structured generation capabilities of large language models (LLMs), focusing on producing valid JSON outputs against a given schema. Despite the widespread use of JSON in integrating language models with programs, there is a lack of comprehensive analysis and benchmarking of these capabilities. We explore various aspects of JSON generation, such as structure understanding, escaping, and natural language description, to determine how to assess and enable LLMs to generate valid responses. Building upon this, we propose SchemaBench features around 40K different JSON schemas to obtain and assess models' abilities in generating valid JSON. We find that the latest LLMs are still struggling to generate a valid JSON string. Moreover, we demonstrate that incorporating reinforcement learning with a Fine-grained Schema Validator can further enhance models' understanding of JSON schema, leading to improved performance. Our models demonstrate significant improvement in both generating JSON outputs and downstream tasks.