# CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following

arXiv: https://arxiv.org/abs/2403.03129 · HTML: https://mlsanity.com/paper/2403.03129/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 58 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Chain-of-Thought Reasoning Without Prompting](https://mlsanity.com/paper/2402.10200/) (2024)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [Offsite-Tuning: Transfer Learning without Full Model](https://mlsanity.com/paper/2302.04870/) (2023)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [An Emulator for Fine-Tuning Large Language Models using Small Language Models](https://mlsanity.com/paper/2310.12962/) (2023)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [Learning Interpretable Style Embeddings via Prompting LLMs](https://mlsanity.com/paper/2305.12696/) (2023)
- [CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models](https://mlsanity.com/paper/2305.16876/) (2023)

## Built on by (most cited first, 30 of 33)

- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models](https://mlsanity.com/paper/2407.06089/) (2024)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [When One LLM Drools, Multi-LLM Collaboration Rules](https://mlsanity.com/paper/2502.04506/) (2025)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models](https://mlsanity.com/paper/2502.15964/) (2025)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding](https://mlsanity.com/paper/2406.12295/) (2024)
- [The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Prompts](https://mlsanity.com/paper/2406.14318/) (2024)
- [CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steering](https://mlsanity.com/paper/2507.04756/) (2025)
- [Step-Back Profiling: Distilling User History for Personalized Scientific Writing](https://mlsanity.com/paper/2406.14275/) (2024)
- [GPT for Games: An Updated Scoping Review (2020-2024)](https://mlsanity.com/paper/2411.00308/) (2024)
- [Towards Building Specialized Generalist AI with System 1 and System 2 Fusion](https://mlsanity.com/paper/2407.08642/) (2024)
- [From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling](https://mlsanity.com/paper/2506.16393/) (2025)
- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [Contextualized Privacy Defense for LLM Agents](https://mlsanity.com/paper/2603.02983/) (2026)
- [FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices](https://mlsanity.com/paper/2410.00362/) (2024)
- [Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks](https://mlsanity.com/paper/2504.17421/) (2025)
- [Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM](https://mlsanity.com/paper/2508.07260/) (2025)
- [A General Pseudonymization Framework for Cloud-Based LLMs: Replacing Privacy Information in Controlled Text Generation](https://mlsanity.com/paper/2502.15233/) (2025)
- [Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions](https://mlsanity.com/paper/2504.15300/) (2025)
- [Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Models](https://mlsanity.com/paper/2512.03056/) (2025)
- [Floe: Federated Specialization for Real-Time LLM-SLM Inference](https://mlsanity.com/paper/2602.14302/) (2026)
- [Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM](https://mlsanity.com/paper/2604.17827/) (2026)
- [Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2509.04459/) (2025)
- [PrivScope: Task-scoped Disclosure Control for Hybrid Agentic Systems](https://mlsanity.com/paper/2605.16630/) (2026)
- [Collaborative LLM Numerical Reasoning with Local Data Protection](https://mlsanity.com/paper/2504.00299/) (2025)
- [Adaptive Backtracking for Privacy Protection in Large Language Models](https://mlsanity.com/paper/2508.06087/) (2025)
- [Auditing and Mitigating Privacy Leakage in Cloud-Edge Collaborative Decoding](https://mlsanity.com/paper/2608.29111/) (2026)

## Abstract (arXiv)

With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.