# Do You Have the Right Scissors? Tailoring Pre-trained Language Models via Monte-Carlo Methods

arXiv: https://arxiv.org/abs/2007.06162 · HTML: https://mlsanity.com/paper/2007.06162/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 5 of 5)

- [Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference](https://mlsanity.com/paper/2508.02193/) (2025)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo](https://mlsanity.com/paper/2504.13139/) (2025)
- [Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling](https://mlsanity.com/paper/2504.05410/) (2025)
- [Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offs](https://mlsanity.com/paper/2112.05702/) (2021)

## Abstract (arXiv)

It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimation problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to under-estimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach. Our code is available at this url.