CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records
Not on the map. Why?
Abstract (arXiv)
Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy-Hardened Evidence Records), a benchmark of expert-validated questions from consumer-finance, clinical, and law-enforcement records. The questions cover common tabular operations and include executable SQL supervision. We evaluate retrieval, prompting, table-specialist, and hybrid symbolic-neural systems under native redaction and surrogate-based evidence restoration. All system families exhibit substantial failures even when supporting records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform effects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transformations of sensitive text affect reasoning over hybrid records.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 7 Sep 2026.
Builds on
20 of 33 references.
References · 20
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
- 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
- 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
- 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
- 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
- 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
- 2022Binding Language Models in Symbolic Languagescited by 224
8 more
- 2021FeTaQA: Free-form Table Question Answeringcited by 174
- 2020Open Question Answering over Tables and Textcited by 163
- 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
- 2024BlendSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebracited by 23
- 2024H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tablescited by 21
- 2024TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Datacited by 19
- 2025Weaver: Interweaving SQL and LLM for Table Reasoningcited by 8
- 2025RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tablescited by 1
Built on by
Not cited yet.