SumRe: Design and Evaluation of a Gist-based Summary Visualization for Incident Reports Triage

Abstract

Incident report triage is a common endeavor in many industry sectors, often coupled with serious public safety implications. For example, at the US Food and Drug Administration (FDA), analysts triage an influx of incident reports to identify previously undiscovered drug safety problems. However, these analysts currently conduct this critical yet error-prone incident report triage using a generic table-based interface, with no formal support. Visualization design, task-characterization methodologies, and evaluation models offer several possibilities for better supporting triage workflows, including those dealing with drug safety and beyond. In this work, we aim to elevate the work of triage through a task-abstraction activity with FDA analysts. Second, we design an alternative gist-based summary of text documents used in triage (SumRe). Third, we conduct a crowdsourced evaluation of SumRe with medical experts. Results of the crowdsourced study with medical experts (n = 20) suggest that SumRe better supports accuracy in understanding the gist of a given report, and in identifying important reports for followup activities. We discuss implications of these results, including design considerations for triage workflows beyond the drug domain, as well as methodologies for comparing visualization-enabled text summaries.