mlsanityData to 7 Oct 2026

Memorization for Good: Encryption with Autoregressive Language Models

arXiv 2305.10445 ↗First posted 15 May 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Over-parameterized neural language models (LMs) can memorize and recite long sequences of training data. While such memorization is normally associated with undesired properties such as overfitting and information leaking, our work casts memorization as an unexplored capability of LMs. We propose the first symmetric encryption algorithm with autoregressive language models (SELM). We show that autoregressive LMs can encode arbitrary data into a compact real-valued vector (i.e., encryption) and then losslessly decode the vector to the original message (i.e., decryption) via random subspace optimization and greedy decoding. While SELM is not amenable to conventional cryptanalysis, we investigate its security through a novel empirical variant of the classic IND-CPA (indistinguishability under chosen-plaintext attack) game and show promising results on security. Our code and datasets are available at https://github.com/OSU-NLP-Group/SELM.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 56 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

20 of 56 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
8 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  5. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  6. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  7. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 72
  8. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28

Built on by

Not cited yet.