Show simple item record

dc.contributor.authorSevastjanova, Ritaen_US
dc.contributor.authorKalouli, Aikaterini-Lidaen_US
dc.contributor.authorBeck, Christinen_US
dc.contributor.authorHauptmann, Hannaen_US
dc.contributor.authorEl-Assady, Mennatallahen_US
dc.contributor.editorBorgo, Ritaen_US
dc.contributor.editorMarai, G. Elisabetaen_US
dc.contributor.editorSchreck, Tobiasen_US
dc.date.accessioned2022-06-03T06:06:12Z
dc.date.available2022-06-03T06:06:12Z
dc.date.issued2022
dc.identifier.issn1467-8659
dc.identifier.urihttps://doi.org/10.1111/cgf.14541
dc.identifier.urihttps://diglib.eg.org:443/handle/10.1111/cgf14541
dc.description.abstractLanguage models, such as BERT, construct multiple, contextualized embeddings for each word occurrence in a corpus. Understanding how the contextualization propagates through the model's layers is crucial for deciding which layers to use for a specific analysis task. Currently, most embedding spaces are explained by probing classifiers; however, some findings remain inconclusive. In this paper, we present LMFingerprints, a novel scoring-based technique for the explanation of contextualized word embeddings. We introduce two categories of scoring functions, which measure (1) the degree of contextualization, i.e., the layerwise changes in the embedding vectors, and (2) the type of contextualization, i.e., the captured context information. We integrate these scores into an interactive explanation workspace. By combining visual and verbal elements, we provide an overview of contextualization in six popular transformer-based language models. We evaluate hypotheses from the domain of computational linguistics, and our results not only confirm findings from related work but also reveal new aspects about the information captured in the embedding spaces. For instance, we show that while numbers are poorly contextualized, stopwords have an unexpected high contextualization in the models' upper layers, where their neighborhoods shift from similar functionality tokens to tokens that contribute to the meaning of the surrounding sentences.en_US
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectCCS Concepts: Human-centered computing --> Visual analytics; Information visualization
dc.subjectHuman centered computing
dc.subjectVisual analytics
dc.subjectInformation visualization
dc.titleLMFingerprints: Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scoresen_US
dc.description.seriesinformationComputer Graphics Forum
dc.description.sectionheadersText and Music
dc.description.volume41
dc.description.number3
dc.identifier.doi10.1111/cgf.14541
dc.identifier.pages295-307
dc.identifier.pages13 pages


Files in this item

Thumbnail
Thumbnail

This item appears in the following Collection(s)

  • 41-Issue 3
    EuroVis 2022 - Conference Proceedings

Show simple item record

Attribution 4.0 International License
Except where otherwise noted, this item's license is described as Attribution 4.0 International License