In this talk, I will focus on resolution of two important subclasses of constrained optimization: bound-constrained problems and linear programming. They are motivated by popular machine learning topics including nonnegative matrix factorization and optimal transport (OT). To resolve the former subclass, I will introduce a two-metric projection method which effectively exploit Hessian information of the objective function. This method inspires several algorithms including a projected Newton-CG equipped with optimal worst-case complexity guarantees, and an adaptive two-metric projection method designed to address l1-norm regularization. For the linear programming formulation of OT, I will discuss random block coordinate descent (RBCD) methods. A direct advantage of these methods is to save memory and we demonstrate its efficiency by comparison with competitors including the classical Sinkhorn algorithm.

The application of Tensor Train (TT) decomposition in machine learning models provides a promising approach to addressing challenges related to model size and computational complexity. TT decomposition, by breaking down high-dimensional weight tensors into smaller, more manageable tensor cores, allows for significant reductions in model size while maintaining performance. This presentation will explore how TT decomposition can be effectively used in different types of models.

TT decomposition is adopted differently in recurrent models, Convolutional Neural Networks (CNN), and Binary Neural Networks (BNN). In recurrent models like Long Short-Term Memory (LSTM), large weight matrices are transformed into smaller, manageable tensor cores, reducing the number of parameters and computational load. For CNNs, TT decomposition targets the convolutional layers, transforming convolutional filters into tensor cores to preserve spatial structure while significantly reducing parameters. In BNNs, TT decomposition is combined with weight binarization, resulting in extremely compact models that retain essential information for accurate predictions even with minimal computational power and memory.

The primary aim of this presentation is to explore the theoretical foundations and practical applications of TT decomposition, demonstrating how this technique optimizes various machine learning models. The findings suggest that TT decomposition can greatly enhance model efficiency and scalability, making it a valuable tool for a wide range of applications.