# Weaver: Foundation Models for Creative Writing

arXiv: https://arxiv.org/abs/2401.17268 · HTML: https://mlsanity.com/paper/2401.17268/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)

## Built on by (most cited first, 27 of 27)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [WritingBench: A Comprehensive Benchmark for Generative Writing](https://mlsanity.com/paper/2503.05244/) (2025)
- [Detoxifying Large Language Models via Knowledge Editing](https://mlsanity.com/paper/2403.14472/) (2024)
- [MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series](https://mlsanity.com/paper/2405.19327/) (2024)
- [PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents](https://mlsanity.com/paper/2402.12326/) (2024)
- [MIO: A Foundation Model on Multimodal Tokens](https://mlsanity.com/paper/2409.17692/) (2024)
- [Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualization](https://mlsanity.com/paper/2408.04112/) (2024)
- [BadActs: A Universal Backdoor Defense in the Activation Space](https://mlsanity.com/paper/2405.11227/) (2024)
- [Sailor: Open Language Models for South-East Asia](https://mlsanity.com/paper/2404.03608/) (2024)
- [OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking](https://mlsanity.com/paper/2501.09751/) (2025)
- [Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs](https://mlsanity.com/paper/2409.11547/) (2024)
- [TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories](https://mlsanity.com/paper/2511.21322/) (2025)
- [PositionID: LLMs can Control Lengths, Copy and Paste with Explicit Positional Awareness](https://mlsanity.com/paper/2410.07035/) (2024)
- [LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information](https://mlsanity.com/paper/2502.02095/) (2025)
- [IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment](https://mlsanity.com/paper/2505.12762/) (2025)
- [Decoupling Content and Expression: Two-Dimensional Detection of AI-Generated Text](https://mlsanity.com/paper/2503.00258/) (2025)
- [SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins](https://mlsanity.com/paper/2410.09362/) (2024)
- [Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs](https://mlsanity.com/paper/2410.14570/) (2024)
- [LLM-driven Constrained Copy Generation through Iterative Refinement](https://mlsanity.com/paper/2504.10391/) (2025)
- [Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01158/) (2024)
- [Arabic Automatic Story Generation with Large Language Models](https://mlsanity.com/paper/2407.07551/) (2024)
- [TaleForge: Interactive Multimodal System for Personalized Story Creation](https://mlsanity.com/paper/2506.21832/) (2025)
- [Anchorless Diversification for Parallel LLM Ideation](https://mlsanity.com/paper/2605.30150/) (2026)
- [MarkSec: Capability-Aware Evaluation of Adversarial Attacks Against LLM Watermarks](https://mlsanity.com/paper/2609.16681/) (2026)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Co-policy: Responsive Human-Robot Co-Creation for Musical Performances](https://mlsanity.com/paper/2606.19914/) (2026)
- [BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences](https://mlsanity.com/paper/2607.27366/) (2026)

## Abstract (arXiv)

This work introduces Weaver, our first family of large language models (LLMs) dedicated to content creation. Weaver is pre-trained on a carefully selected corpus that focuses on improving the writing capabilities of large language models. We then fine-tune Weaver for creative and professional writing purposes and align it to the preference of professional writers using a suit of novel methods for instruction data synthesis and LLM alignment, making it able to produce more human-like texts and follow more diverse instructions for content creation. The Weaver family consists of models of Weaver Mini (1.8B), Weaver Base (6B), Weaver Pro (14B), and Weaver Ultra (34B) sizes, suitable for different applications and can be dynamically dispatched by a routing agent according to query complexity to balance response quality and computation cost. Evaluation on a carefully curated benchmark for assessing the writing capabilities of LLMs shows Weaver models of all sizes outperform generalist LLMs several times larger than them. Notably, our most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes. Moreover, Weaver natively supports retrieval-augmented generation (RAG) and function calling (tool usage). We present various use cases of these abilities for improving AI-assisted writing systems, including integration of external knowledge bases, tools, or APIs, and providing personalized writing assistance. Furthermore, we discuss and summarize a guideline and best practices for pre-training and fine-tuning domain-specific LLMs.