Machine Learning Supported Interactive Visualization of Hybrid 3D and 2D Data for the Example of Plant Cell Lineage Specification

Abstract

As computer graphics technologies develop, spatial data can be better visualized in the 3D environment so that viewers can observe 3D shapes and positions clearly. Meanwhile, 2D abstract visualizations can present summarized information, visualize additional data, and control the 3D view. Combining these two parts in one interface can assist people in finishing complicated tasks, especially in scientific domains, though there is a lack of design guidelines for the interaction. Generally, experts need to analyze large amounts of scientific data to finish challenging tasks. For example, in the biological field, biologists need to build the hierarchy tree for an embryo with more than 200 cells. In this case, manual work can be time-consuming and tedious, and machine learning algorithms have the potential to alleviate some of the tedious manual processes to serve as the basis for experts. These predictions, however, contain hierarchical and multi-layer information, and it is essential to visualize them sequentially and progressively so that experts can control their viewing pace and validation. Also, 3D and 2D representations, together with machine learning predictions, need to be visually and interactively connected in the system.In this thesis, we worked on the cell lineage problem for plant embryos as an example to investigate a visualization system and its interaction design that makes use of combinations of 3D and 2D representations as well as visualizations for machine learning. We first investigated the 3D selection interaction techniques for the plant embryo. The cells in a plant embryo are tightly packed together, without any space in between. Traditional techniques can hardly deal with such an occlusion problem. We conducted a study to evaluate three different selection techniques and found out that the combination of the Explosion Selection technique and the List Selection technique works well for people to get access to and observe plant cells in an embryo. These techniques can also be extended to other similar densely packed 3D data. Second, we explored the visualization and interaction de-sign to combine the 3D visualizations of a plant embryo with its associated 2D hierarchy tree. We designed a system with such combinations for biologists to examine the plant cells and record the development history in the hierarchy tree. We support the hierarchy building in two directions, both constructing the history top-down using the lasso selection in a 3D environment and bottom-up as the traditional workflow does in the hierarchy tree. We also added a neural network model to give predictions about the assignments for biologists to start with. We conducted an evaluation with biologists, which showed that both 3D and 2D representations help with making decisions, and the tool can inspire insights for them. One main drawback was that the performance of the machine learning model was not ideal. Thus, to assist the process and enhance the model performance, in an improved version of our system, we trained five different ML models and visualized the predictions and their associated uncertainty. We performed a study, and the results indicated that our designed ML representations are easy to understand and that the new tool can effectively improve the efficiency of assigning the cell lineage.