Stanford ICME Lecture on Deep Learning

Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including production quality, pre-trained models and smaller models trained from scratch. Empirical and theoretical results indicate that the DNN training process itself implements a form of self-regularization, evident in the empirical spectral density (ESD) of DNN layer matrices. To understand this, we provide a phenomenology to identify 5 1 Phases of Training, corresponding to increasing amounts of i