Date of Award

Summer 8-2025

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Mathematics & Statistics

Program/Concentration

Computational and Applied Mathematics

Committee Director

Yuesheng Xu

Committee Member

Yan Peng

Committee Member

Michael Pokojovy

Committee Member

Guohui Song

Committee Member

Frank Liu

Abstract

Multi-Grade Deep Learning (MGDL) is a training framework that incrementally builds deep neural networks. It does this by dividing the training process into multiple “grades,” where each grade sequentially trains a shallow neural network to learn the residue from the previous one, using the outputs of prior grades as input. This approach progresses from shallow to deep architectures. This dissertation offers a comprehensive theoretical and numerical analysis of the MGDL methodology.

We first demonstrate that MGDL can effectively learn target functions within the sum-composition learning format. In this context, MGDL approximates high-frequency components by composing multiple low-frequency functions. This unique capability explains how MGDL addresses the spectral bias commonly observed in standard deep neural networks, which typically struggle to learn high-frequency features. Extensive numerical experiments confirm that MGDL significantly improves the learning of these high-frequency components in target functions.

Next, we analyze the gradient descent method for training deep neural networks and highlight MGDL’s advantages over standard architectures. Specifically, we show that when each “grade” in MGDL consists of a single hidden layer with a piecewise linear activation function, it transforms a highly nonconvex optimization problem into a sequence of convex subproblems. This transformation provides a theoretical explanation for MGDL’s improved trainability.

We then apply MGDL to image reconstruction tasks, proposing a proximal gradient method to solve the resulting optimization problem and providing a convergence analysis of the algorithm. Experimental results indicate that MGDL achieves more stable training compared to standard deep neural networks. To understand this enhanced stability, we conduct an eigenvalue analysis of the iterative matrices arising from gradient descent. This analysis reveals that training stability is governed by the distribution of these eigenvalues. Numerical experiments show that standard deep neural networks often produce eigenvalues outside the convergence region, while MGDL consistently maintains eigenvalues within it, thereby explaining its superior stability

Rights

In Copyright. URI: http://rightsstatements.org/vocab/InC/1.0/ This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s).

DOI

10.25777/df7n-0h32

ISBN

9798293841516

Download

Share

COinS