Progress on the Interplay of Machine Learning and Optimization

Abstract

Machine learning and optimization have been playing significant roles in the world. Despite the remarkable advancements in these fields, various crucial problems remain unsolved. In this thesis, we address some of these problems by exploring the interplay of machine learning and optimization. In the first part of this thesis, we utilize optimization tools to address two practically important and critical topics in machine learning: interpretability of machine learning models, and improving data for prediction. In Chapter 2 and 3, we focus on improving the interpretability of machine learning models. In particular, Chapter 2 presents an efficient algorithm for training high-quality Nonlinear Oblique Classification Trees using gradient descent. We demonstrate on real-world datasets this is an effective approach. In Chapter 3, we develop an optimization approach to train low depth (up to depth 8) classification trees with hyperplanes to closely approximate neural networks. We also incorporate sparsity in the hyperplanes of the trees. In this way, we contribute in increasing the interpretability of neural networks. Computational results on real-world datasets with different sizes of neural networks show the effectiveness of our algorithm. In Chapter 4, we propose an integer optimization method to improve class-imbalanced data. Our method undersamples the majority class and performs better than existing methods on real-world imbalanced datasets. In the second part of the thesis, we explore the direction of applying machine learning to optimization. In Chapter 5, we show that optimization methods can significantly benefit from a machine learning treatment. We develop a model-based trust-region method for derivative-free optimization problems under noise. Our method, which uses robust and sparse regression to build models of functions, is much more robust and has higher scalability than existing methods.