Semantic Image Abstraction using Panoptic Segmentation for Robotic Painting

Abstract

We propose a comprehensive pipeline for generating adaptable image abstractions from input pictures, tailored explicitly for robotic painting tasks. Our pipeline addresses several key objectives, including the ability to paint from background to foreground, maintain fine details, capture structured regions accurately, and highlight important objects. To achieve this, we employ a panoptic segmentation network to predict the semantic class membership for each pixel in the image. This step provides us with a detailed understanding of the object categories present in the scene. Building upon the semantic segmentation results, we combine them with a color-based image over-segmentation technique. This process partitions the image into monochromatic regions, each corresponding to a specific semantic object. Next, we construct a hierarchical tree based on the segmentation results, which allows us to merge adjacent regions based on their color difference and semantic class. We take care to ensure that shapes belonging to different semantic objects are not merged together. We iteratively perform adjacency merging until no further combinations are possible, resulting in a refined hierarchical shape tree. To obtain the desired image abstraction, we filter the hierarchical shape tree by examining factors such as color differences, relative sizes, and the layering within the hierarchy of each region in relation to their parent regions. By employing this approach, we can preserve fine details, apply local filtering operations, and effectively combine regions with structured shapes. This results in image abstractions well-suited for robotic painting applications and artistic renderings.