An HMD-based Mixed Reality System for Avatar-Mediated Remote Collaboration with Bare-hand Interaction

Abstract

We present a novel system for mixed reality based remote collaboration system, which enables a local user to interact and collaborate with another user from remote space using natural hand motion. Unlike conventional system where the remote user appears only inside the screen, our system is able to summon the remote user into the local space, which appears as a virtual avatar in the real world view seen by the local user. To support our avatar-mediated remote collaboration concept, we derive a systematic framework design that consists of the hardware and software configuration with various devices. We explore novel techniques for calibrating and managing the coordinate system in asymmetric setup, sensor fusion between devices and generating human-like motion for the avatar. For validating our proposal, we implemented a proof-of-concept prototype using off-the-shelf hardware and report the experimental results. We believe that our system overcomes not only several limitations of previous systems but also creates new possibilities in remote collaboration domain.