mlsanityData to 7 Oct 2026

Tangent Transformers for Composition, Privacy and Removal

arXiv 2307.08122 ↗First posted 16 Jul 2023cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce Tangent Attention Fine-Tuning (TAFT), a method for fine-tuning linearized transformers obtained by computing a First-order Taylor Expansion around a pre-trained initialization. We show that the Jacobian-Vector Product resulting from linearization can be computed efficiently in a single forward pass, reducing training and inference cost to the same order of magnitude as its original non-linear counterpart, while using the same number of parameters. Furthermore, we show that, when applied to various downstream visual classification tasks, the resulting Tangent Transformer fine-tuned with TAFT can perform comparably with fine-tuning the original non-linear network. Since Tangent Transformers are linear with respect to the new set of weights, and the resulting fine-tuning loss is convex, we show that TAFT enjoys several advantages compared to non-linear fine-tuning when it comes to model composition, parallel training, machine unlearning, and differential privacy. Our code is available at: https://github.com/tianyu139/tangent-model-composition

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 16 Jul 2023.

Builds on

36 of 51 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2019Machine Unlearningcited by 1,020
  12. 2021Learning to Prompt for Continual Learningcited by 938
24 more
  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  2. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  3. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  5. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
  6. 2020Mixed-Privacy Forgetting in Deep Networkscited by 158
  7. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  8. 2020A Survey on Negative Transfercited by 120
  9. 2022Fusing finetuned models for better pretrainingcited by 110
  10. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  11. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  12. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  13. 2022Mixed Differential Privacy in Computer Visioncited by 53
  14. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  15. 2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
  16. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 35
  17. 2020Gradients as Features for Deep Representation Learningcited by 30
  18. 2023SAFE: Machine Unlearning With Shard Graphscited by 28
  19. 2020LQF: Linear Quadratic Fine-Tuningcited by 25
  20. 2021Differentially Private SGD with Non-Smooth Lossescited by 22
  21. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  22. 2023`A-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Promptingcited by 17
  23. 2023AI Model Disgorgement: Methods and Choicescited by 9
  24. 2022Integral Continual Learning Along the Tangent Vector Field of Taskscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  3. 2024CPR: Retrieval Augmented Generation for Copyright Protectioncited by 22
  4. 2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14
  5. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  6. 2025LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agentscited by 6
  7. 2025PICASO: Permutation-Invariant Context Composition with State Space Modelscited by 4
  8. 2025Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMscited by 4
  9. 2025Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planningcited by 4
  10. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  11. 2025DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentationcited by 3
  12. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
1 more
  1. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 1