Sketching User Interfaces with Visual Patterns

Abstract

This paper presents an approach to layout static components of user interfaces as hand-drawn compositions of simple geometric shapes, using sketch recognition and visual languages. The system uses a visual grammar built from drawings collectedfrom users. We tried to understand how people sketch interfaces and what combinations of shapes they are more likely to used when sketching widgets. From there we implemented a prototype system, for creating user interfaces through hand-drawn geometric shapes, identified by a gesture recognizer. This prototype generates a Java interface, whose layout can be beautified using an a posteriori set of grammar rufes ( e.g. to align and group objects, etc.). We have conducted usability assessments with ten users to compare our approach with a commercial system (JBuilder). Besides a measurable speed advantage in drawing interfaces, users found our system more comfortable, satisfactory and efficient to use than the commercial product, as demonstrated by post-experiment questionnaires.