JavaSketchIt2 – Using Relational Adjacency Grammars for Visual Syntax Parsing

Abstract

This paper presents a project which expands on JavaSketchit [8], which parsed sketches of user interfaces, iden- tifying gestures by using CALI [6], and then used topological relations and relational constraints [4] to recog- nize pairs of gestures as widgets in a Java graphical user interface. JavaSketchit was able to export the func- tional source code for the prototype in Java. Our approach allows users to write down their own visual lan- guage and removes the existing limitation of one pair of gestures per widget. The resulting parser became ap- plicable in different contexts besides designing user interfaces. The result is a functional implementation of a parser designed to handle Relational Adjacency Grammars customized by the user..