Zooming On All Actors: Automatic Focus+Context Split Screen Video Generation

Abstract

Stage performances can be easily captured using a high resolution camera but these are often difficult to watch because actor faces are too small. We present a novel approach to create a split screen video that incorporates both the context as well as the close-up details of the actors. Our system takes as input the static recording of a stage performance and tracking information about the actor positions, and generates a video with a wide master shot and a set of close-ups of all identified actors and hence showing a focus+context view that shows both the overall action as well as the details of actor faces. The key to our approach is to compute these camera motions such that they are cinematically valid close-ups and to ensure that the set of views of the different actors are properly coordinated and presented. The close-up views are created as virtual camera movements by applying panning, cropping and zooming to the source video. We pose the computation of camera motions as convex optimization that creates detailed views and smooth movements, subject to cinematic constraints such as not cutting faces with the edge of the frame. Additional constraints allow for the interaction amongst the close up views of each actor, causing them to merge seamlessly when actors are close. Generated views are then placed in a layout that preserves the spatial relationships between actors. We demonstrate our results on a variety of video sequences from theatre and dance performances.