mlsanityData to 14 Oct 2026

StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation

arXiv 2504.04373 ↗First posted 6 Apr 2025cs.CL cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Prompt Recovery, reconstructing prompts from the outputs of large language models (LLMs), has grown in importance as LLMs become ubiquitous. Most users access LLMs through APIs without internal model weights, relying only on outputs and logits, which complicates recovery. This paper explores a unique prompt recovery task focused on reconstructing prompts for style transfer and rephrasing, rather than typical question-answering. We introduce a dataset created with LLM assistance, ensuring quality through multiple techniques, and test methods like zero-shot, few-shot, jailbreak, chain-of-thought, fine-tuning, and a novel canonical-prompt fallback for poor-performing cases. Our results show that one-shot and fine-tuning yield the best outcomes but highlight flaws in traditional sentence similarity metrics for evaluating prompt recovery. Contributions include (1) a benchmark dataset, (2) comprehensive experiments on prompt recovery strategies, and (3) identification of limitations in current evaluation metrics, all of which advance general prompt recovery research, where the structure of the input prompt is unrestricted.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

28 of 39 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Mistral 7Bcited by 3,164
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  11. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  12. 2023Improving Text Embeddings with Large Language Modelscited by 303
16 more
  1. 2023The False Promise of Imitating Proprietary LLMscited by 263
  2. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  3. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
  4. 2023Copyright Violations and Large Language Modelscited by 108
  5. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  6. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
  7. 2023Language Model Inversioncited by 68
  8. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 65
  9. 2024Prompt Stealing Attacks Against Large Language Modelscited by 40
  10. 2023Better patching using LLM prompting, via Self-Consistencycited by 29
  11. 2019CANZSL: Cycle-Consistent Adversarial Networks for Zero-Shot Learning from Natural Languagecited by 23
  12. 2022Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networkscited by 21
  13. 2024DORY: Deliberative Prompt Recovery for LLMcited by 7
  14. 2024Was it Slander? Towards Exact Inversion of Generative Language Modelscited by 3
  15. 2024Uncovering Hidden Intentions: Exploring Prompt Recovery for Deeper Insights into Generated Textscited by 2
  16. 2024Advancing Prompt Recovery in NLP: A Deep Dive into the Integration of Gemma-2b-it and Phi2 Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025GPT-FT: An Efficient Automated Feature Transformation Using GPT for Sequence Reconstruction and Performance Enhancementcited by 1
  2. 2025SAVeD: A First-Person Social Media Video Dataset for ADAS-equipped vehicle Near-Miss and Crash Event Analysescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles
  2. 2024Extracting Prompts by Inverting LLM Outputs
  3. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  4. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  5. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
  6. 2024Distilling Text Style Transfer With Self-Explanation From LLMs
  7. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression
  8. 2025A Survey on Prompt Tuning