mlsanityData to 14 Oct 2026

HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

arXiv 2607.27379 ↗First posted 29 Jul 2026cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16 benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 38 of its 56 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

38 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement21 Jan
  2. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personas28 Jun

Other references · 36

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
24 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 683
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  5. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  7. 2021Efficient Attentions for Long Document Summarizationcited by 345
  8. 2016FastText.zip: Compressing text classification modelscited by 335
  9. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  10. 2023Large Language Models as Tool Makerscited by 303
  11. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 216
  12. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  13. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  14. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  15. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  16. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 70
  17. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  18. 2023TabuLa: Harnessing Language Models for Tabular Data Synthesiscited by 42
  19. 2024Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptationcited by 27
  20. 2023DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrappingcited by 16
  21. 2025Instruction-Tuning Data Synthesis from Scratch via Web Reconstructioncited by 8
  22. 2025Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Promptscited by 5
  23. 2025From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Groundingcited by 5
  24. 2025DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilitiescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
  2. 2024LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis
  3. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models
  4. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  5. 2026BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
  6. 2024Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Research
  7. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers
  8. 2025A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesis