mlsanityData to 14 Oct 2026

JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuning

arXiv 2310.02953 ↗First posted 4 Oct 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Instruction tuning is vital for enhancing the performance of large language models (LLMs), but existing text-to-text methods, referred to as TextTuning, struggle with issues such as generalization, robustness, and controllability due to their lack of explicit task structures. We introduce JsonTuning, a structure-to-structure approach that uses JSON structures to represent tasks. This method improves generalization by clarifying task elements and their relations, boosts robustness by minimizing ambiguity, and enhances controllability by allowing precise control over outputs. We conduct an extensive comparative analysis between JsonTuning and TextTuning using various language models and benchmarks. Our findings reveal that JsonTuning consistently surpasses TextTuning in terms of performance, robustness, and controllability across different scenarios. By overcoming the limitations of TextTuning, JsonTuning demonstrates significant potential for developing more effective and reliable LLMs capable of handling diverse scenarios.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 56 of its 93 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

56 of 93 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023LIMA: Less Is More for Alignmentcited by 1,100
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  14. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  15. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  16. 2023Instruction Tuning with GPT-4cited by 686
  17. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  18. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  19. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  20. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  21. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  22. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  23. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  24. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  25. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 170
  26. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 163
  27. 2023Training Language Models with Language Feedback at Scalecited by 128
  28. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  29. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 104
  30. 2023Symbol tuning improves in-context learning in language modelscited by 75
  31. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 66
  32. 2018A Hierarchical Framework for Relation Extraction with Reinforcement Learningcited by 49
  33. 2015Relation Classification via Recurrent Neural Networkcited by 46
  34. 2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45
  35. 2023Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learningcited by 41
  36. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  37. 2022PHEE: A Dataset for Pharmacovigilance Event Extraction from Textcited by 37
  38. 2023Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Followingcited by 33

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 41
  2. 2023Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Trackingcited by 5
  3. 2024Focus On This, Not That! Steering LLMs with Adaptive Feature Specificationcited by 5
  4. 2025eC-Tab2Text: Aspect-Based Text Generation from e-Commerce Product Tablescited by 2
  5. 2024EXCGEC: A Benchmark for Edit-Wise Explainable Chinese Grammatical Error Correctioncited by 1
  6. 2025Low-Resource Fine-Tuning for Multi-Task Structured Information Extraction with a Billion-Parameter Instruction-Tuned Modelcited by 0