Sign Language Learning System with Concurrent Shared Avatar Hand in a Virtual Environment: Psychological Evaluation

Abstract

We aimed to develop a sign language learning system using virtual reality to improve learning motivation. Hand movements for twenty words consisted of three letters were recorded with a hand motion capture system (model hand). In the learning system, the participant was asked to mimic the model hand movement while looking at both the model hand and the ''own hand'' in a head mounted display (HMD) with the hand motion capture. The ''own hand'' avatar was either of the real own hand or the shared hand motion, which was made by averaging the participant's hand and the model hand movements. The model hand was presented either in the opposite or same direction as the participant. Participants rated the usability of the system in 2 x 2 (own/shared hand x opposite/same direction) experimental blocked design. We found that the shared hand avatar and the same direction presentation were better than the own hand and the opposite direction presentation, respectively. Thus, the proposed shared hand avatar system with the HMD and hand motion capture could improve sign language learning.