Mobile Vision-Based Sketch Recognition with SPARK

Abstract

The sketch community has, over the past years, developed a powerful arsenal of recognition capabilities and interaction methods. Unfortunately, many people who could benefit from these systems lack pen capture hardware and are stuck drawing diagrams on traditional surfaces like paper or whiteboards. In this paper we explore bringing the benefits of sketch capture and recognition to traditional surfaces through a common smart-phone with the Sketch Practically Anywhere Recognition Kit (SPARK), a framework for building mobile, image-based sketch recognition applications. Naturally, there are several challenges that come with recognizing hand-drawn diagrams from a single image. Image processing techniques are needed to isolate marks from the background surface due to variations in lighting and surface wear. Further, since static images contain no notion of how the original diagram was drawn, we employ bitmap thinning and stroke tracing to transform the ink into the abstraction of strokes commonly used by modern sketch recognition algorithms. Since the timing data between points in each stroke are not present, recognition must remain robust to variability in both perceived drawing speed and even coarse ordering between points. We have evaluated Rubine's recognizer in an effort to quantify the impact of timing information on recognition, and our results show that accuracy can remain consistent in spite of artificially traced stroke data. As evidence of our techniques, we have implemented a mobile app in SPARK that captures images of Turing machine diagrams drawn on paper, a whiteboard, or even a chalkboard, and through sketch recognition techniques, allows users to simulate the recognized Turing machine on their phones.