Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques

Large language models have transformed natural language processing, yet supervised fine-tuning (SFT) remains computationally intensive. This paper formally proves that capabilities acquired through SFT can be approximated by a base transformer model using inference-time techniques, specifically in-context learning (ICL), without altering model parameters, under idealized assumptions including unbounded computational resources and access to the fine-tuning dataset. We extend these results to practical scenarios with finite context lengths and partial dataset access. For text generation tasks with fixed output length l, datasets of size Oleft( fracm Vvarepsilon2 log fracmδ right) or, with bounded context, Oleft( fracl log Vvarepsilon2 log frac1δ right) suffice to approximate fine-tuned behavior across m contexts within error varepsilon, where V is the vocabulary size and δ is the failure probability. For linear classification, datasets of size Oleft( fracdvarepsilon right) or, with fixed context, Oleft( frac1varepsilon2 log frac1δ right) are sufficient, where d is the input dimension. Grounded in the Turing completeness of transformers, these results provide a theoretical foundation for resource-efficient deployment of large language models, with practical techniques like retrieval-augmented generation bridging theory to real-world applications.