mlsanityData to 14 Oct 2026

From Latent Space to Jacobian Space: Measuring, Evading, and Training Against Safety-Content Accessibility

arXiv 2610.04316 ↗First posted 3 Oct 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Output-only safety monitoring sees only the end of a model's computation, yet the model computes its answer before emitting it: what it is internally poised to say is safety-critical. Jacobian-space (J-space) readouts, linear maps from hidden states to the output vocabulary via the model's input-output Jacobian, have been proposed as a window on behaviorally accessible internal content, and a first safety protocol (JADR) showed that danger recognition is readable there. What remains unknown is how this accessible content relates to the latent-space safety geometry studied by representation engineering, and how safety training shapes it. We introduce two quantitative bridges: (i) a transport-amplification profile Aℓ, measuring how strongly a latent safety direction is carried toward output space by each layer's Jacobian, and (ii) a paired base-vs-tuned protocol that attributes accessibility to training provenance. Across four small model pairs (135M to 1.5B, three lens-fit seeds), tuned checkpoints concentrate recognition in upper-middle layers, and at 0.5B DPO installs refusal while J-space recognition drops to chance: training can widen the accessibility gap exactly where behavioral safety looks best. A deployment audit closes the loop: monitor-aware GCG-style suffixes suppress the prompt-side monitor at zero behavioral cost at every scale; only a learned monitor rung resists its own adaptive re-attack; the training-time defense fails its re-attack at every penalty weight; steering shows the amplification profile is descriptive, not causal; and continuous-prefix optimization fails where discrete search succeeds. Safety monitoring, training, and evaluation must operate on accessibility itself, not on outputs alone.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

33 of 44 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
21 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  4. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  6. 2024A StrongREJECT for Empty Jailbreakscited by 385
  7. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  8. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  9. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  10. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  11. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  12. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  13. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
  14. 2025Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activationscited by 10
  15. 2026Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Controlcited by 9
  16. 2024Applying Refusal-Vector Ablation to Llama 3.1 70B Agentscited by 6
  17. 2025Equivalent Linear Mappings of Large Language Modelscited by 4
  18. 2025SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifoldscited by 2
  19. 2026Silent Alarm: A J-Space Protocol for Comparing Danger Recognition Across Models and Quantization Levelscited by 2
  20. 2026Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokenscited by 1
  21. 2026Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliterationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Safety Geometry Collapses: Fine-Tuning Vulnerabilities in Agentic Guard Models
  2. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis
  3. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
  4. 2026Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Models
  5. 2026DANGERLENS: Probing Dangerous Representations Hidden by Safety Alignment
  6. 2025AnySafe: Adapting Latent Safety Filters at Runtime via Safety Constraint Parameterization in the Latent Space
  7. 2025Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models
  8. 2025Safety Certification in the Latent space using Control Barrier Functions and World Models