Visualizing Attention in Transformer-Based Language Representation Models
Details
Florence, Italy. Date of Talk: 2019-07-28
Speakers
Jesse Vig
Event
Visualizing Attention in Transformer-Based Language Representation Models
We present an open-source tool for visualizing self-attention in Transformer-based language representation models. The tool extends earlier work by visualizing attention at three levels of granularity: the attention-head level, the model level, and the neuron level. We show how each of these views can help to interpret the model, and we demonstrate the tool on BERT and OpenAI GPT-2. We also present three use cases for analyzing GPT-2: detecting model bias, identifying recurring patterns, and linking neurons to model behavior.
Additional information
Focus Areas
Our work is centered around a series of Focus Areas that we believe are the future of science and technology.
Licensing & Commercialization Opportunities
We’re continually developing new technologies, many of which are available for Commercialization.
News
PARC scientists and staffers are active members and contributors to the science and technology communities.