Talk2Hand: Knowledge Board Interaction in Augmented Reality Easing Analysis with Machine Learning Assistants

Abstract

Analysts now often use machine learning (ML) assistants, but find them difficult to use, since most have little ML expertise. Talk2Hand improves the usability of ML assistants by supporting interaction with them using knowledge boards, which intuitively show association, visually aid human recall, and offer natural interaction that eases improvement of displayed associations and addition of new data into emerging models. Knowledge boards are familiar to most and studied by analytics researchers, but not in wide use, because of their large size and the challenges of using them for several projects simultaneously. Talk2Hand uses augmented reality to address these shortcomings, overlaying large but virtual knowledge boards onto typical analyst offices, and enabling analysts to switch easily between different knowledge boards. This paper describes our Talk2Hand prototype.