Visualizing Optimizers using Chebyshev Proxies and Fatou Sets

Abstract

With recent advances in optimization many different optimization approaches have been proposed, especially regarding the optimization of weights for neural networks. However, comparing these approaches in a visually succinct and intuitive manner is difficult to do, especially without relying on simplified toy examples that may not be representative. In this paper, we present a visualization toolkit using a modified variant of Fatou sets of functions in the complex domain to directly visualize the convergence behavior of an optimizer across a large range of input values. Furthermore, we propose an approach of generating test functions based on polynomial Chebyshev proxies, with polynomial degrees up to 11217, and a modification of these proxies to yield functions that are strictly positive with known global minima, i.e., roots. Our proposed toolkit is provided as a cross platform open source framework in C++ using OpenMP for parallelization. Finally, for menomorphic functions the process generates visually interesting fractals, which might also be interesting from an artistic standpoint.