# SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces

arXiv: https://arxiv.org/abs/2511.06778 · HTML: https://mlsanity.com/paper/2511.06778/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 34)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [CodeS: Towards Building Open-source Language Models for Text-to-SQL](https://mlsanity.com/paper/2402.16347/) (2024)
- [C3: Zero-shot Text-to-SQL with ChatGPT](https://mlsanity.com/paper/2307.07306/) (2023)
- [CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL](https://mlsanity.com/paper/2410.01943/) (2024)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites](https://mlsanity.com/paper/2010.02840/) (2020)
- [Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://mlsanity.com/paper/2401.12954/) (2024)
- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [Aligner: Efficient Alignment by Learning to Correct](https://mlsanity.com/paper/2402.02416/) (2024)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2406.09136/) (2024)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL](https://mlsanity.com/paper/2411.08599/) (2024)
- [Preference Optimization for Reasoning with Pseudo Feedback](https://mlsanity.com/paper/2411.16345/) (2024)
- [An Engorgio Prompt Makes Large Language Model Babble on](https://mlsanity.com/paper/2412.19394/) (2024)
- [MPO: Multilingual Safety Alignment via Reward Gap Optimization](https://mlsanity.com/paper/2505.16869/) (2025)
- [TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring](https://mlsanity.com/paper/2403.15879/) (2024)
- [CoE-SQL: In-Context Learning for Multi-Turn Text-to-SQL with Chain-of-Editions](https://mlsanity.com/paper/2405.02712/) (2024)
- [ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL](https://mlsanity.com/paper/2412.10138/) (2024)
- [PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing](https://mlsanity.com/paper/2407.16318/) (2024)
- [Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL](https://mlsanity.com/paper/2502.11656/) (2025)

## Built on by (most cited first, 4 of 4)

- [Natural Language Interfaces for Spatial and Temporal Databases: A Comprehensive Overview of Methods, Taxonomy, and Future Directions](https://mlsanity.com/paper/2603.23375/) (2026)
- [Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents](https://mlsanity.com/paper/2606.26627/) (2026)
- [Policy-Conditioned Constrained Decoding for Column-Level Access Control in Text-to-SQL](https://mlsanity.com/paper/2607.12341/) (2026)
- [TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis](https://mlsanity.com/paper/2608.24145/) (2026)

## Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has driven significant progress in Natural Language Interface to Database (NLIDB). However, the widespread adoption of LLMs has raised critical privacy and security concerns. During interactions, LLMs may unintentionally expose confidential database contents or be manipulated by attackers to exfiltrate data through seemingly benign queries. While current efforts typically rely on rule-based heuristics or LLM agents to mitigate this leakage risk, these methods still struggle with complex inference-based attacks, suffer from high false positive rates, and often compromise the reliability of SQL queries. To address these challenges, we propose textscSafeNlidb, a novel privacy-security alignment framework for LLM-based NLIDB. The framework features an automated pipeline that generates hybrid chain-of-thought interaction data from scratch, seamlessly combining implicit security reasoning with SQL generation. Additionally, we introduce reasoning warm-up and alternating preference optimization to overcome the multi-preference oscillations of Direct Preference Optimization (DPO), enabling LLMs to produce security-aware SQL through fine-grained reasoning without the need for human-annotated preference data. Extensive experiments demonstrate that our method outperforms both larger-scale LLMs and ideal-setting baselines, achieving significant security improvements while preserving high utility. WARNING: This work may contain content that is offensive and harmful!