# Understanding Hidden Memories of Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1710.10777 · HTML: https://mlsanity.com/paper/1710.10777/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2017.

## Built on by (most cited first, 30 of 44)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [Is Sparse Attention more Interpretable?](https://mlsanity.com/paper/2106.01087/) (2021)
- [The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations](https://mlsanity.com/paper/2212.11737/) (2022)
- [Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment](https://mlsanity.com/paper/2202.02838/) (2022)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [A Unified Understanding of Deep NLP Models for Text Classification](https://mlsanity.com/paper/2206.09355/) (2022)
- [A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks](https://mlsanity.com/paper/2102.01792/) (2021)
- [Diagnosing Ensemble Few-Shot Classifiers](https://mlsanity.com/paper/2206.04372/) (2022)
- [GenNI: Human-AI Collaboration for Data-Backed Text Generation](https://mlsanity.com/paper/2110.10185/) (2021)
- [Towards Visual Explainable Active Learning for Zero-Shot Classification](https://mlsanity.com/paper/2108.06730/) (2021)
- [Learning-From-Disagreement: A Model Comparison and Visual Analytics Framework](https://mlsanity.com/paper/2201.07849/) (2022)
- [Understanding Large Language Model Behaviors through Interactive Counterfactual Generation and Analysis](https://mlsanity.com/paper/2405.00708/) (2024)
- [JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models](https://mlsanity.com/paper/2404.08793/) (2024)
- [Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles](https://mlsanity.com/paper/2103.10229/) (2021)
- [A Visual Analytics System for Improving Attention-based Traffic Forecasting Models](https://mlsanity.com/paper/2208.04350/) (2022)
- [ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Dataset](https://mlsanity.com/paper/2208.08010/) (2022)
- [DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction](https://mlsanity.com/paper/2303.01576/) (2023)
- [Visualizing and Explaining Language Models](https://mlsanity.com/paper/2205.10238/) (2022)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Understanding Spending Behavior: Recurrent Neural Network Explanation and Interpretation](https://mlsanity.com/paper/2109.11871/) (2021)
- [Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles](https://mlsanity.com/paper/2208.09665/) (2022)
- [Designing a Direct Feedback Loop between Humans and Convolutional Neural Networks through Local Explanations](https://mlsanity.com/paper/2307.04036/) (2023)
- [A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecasting](https://mlsanity.com/paper/2307.15830/) (2023)
- [Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research](https://mlsanity.com/paper/2308.06290/) (2023)
- [A Review of Hybrid and Ensemble in Deep Learning for Natural Language Processing](https://mlsanity.com/paper/2312.05589/) (2023)
- [EL-VIT: Probing Vision Transformer with Interactive Visualization](https://mlsanity.com/paper/2401.12666/) (2024)
- [CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Models](https://mlsanity.com/paper/2307.12382/) (2023)
- [Abstracted Trajectory Visualization for Explainability in Reinforcement Learning](https://mlsanity.com/paper/2402.07928/) (2024)
- [JailbreakHunter: A Visual Analytics Approach for Jailbreak Prompts Discovery from Large-Scale Human-LLM Conversational Datasets](https://mlsanity.com/paper/2407.03045/) (2024)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have been successfully applied to various natural language processing (NLP) tasks and achieved better results than conventional methods. However, the lack of understanding of the mechanisms behind their effectiveness limits further improvements on their architectures. In this paper, we present a visual analytics method for understanding and comparing RNN models for NLP tasks. We propose a technique to explain the function of individual hidden state units based on their expected response to input texts. We then co-cluster hidden state units and words based on the expected response and visualize co-clustering results as memory chips and word clouds to provide more structured knowledge on RNNs' hidden states. We also propose a glyph-based sequence visualization based on aggregate information to analyze the behavior of an RNN's hidden state at the sentence-level. The usability and effectiveness of our method are demonstrated through case studies and reviews from domain experts.