東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Learning in Large Neural Networks /

Bordelon, Blake,

Linked to FindBook

Google Book

Amazon

博客來

Learning in Large Neural Networks /

Record Type:	Electronic resources : Monograph/item
Title/Author:	Learning in Large Neural Networks // Blake Bordelon.
Author:	Bordelon, Blake,
Description:	1 electronic resource (1396 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-12, Section: B.
Contained By:	Dissertations Abstracts International86-12B.
Subject:	Applied mathematics. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=32041386
ISBN:	9798280710436

Learning in Large Neural Networks /
Bordelon, Blake,

Learning in Large Neural Networks /Blake Bordelon. - 1 electronic resource (1396 pages)

Source: Dissertations Abstracts International, Volume: 86-12, Section: B.

In this thesis, I will summarize my recent works on theoretical frameworks for learning, generalization, scaling limits, and scaling laws of large neural networks. In the first part of the thesis, we will examine the kinds of limits attained by training randomly initialized networks. The two limits of primary interest are the infinite width and infinite depth feature-learning limits.The large width limit will take the form of a dynamical mean field theory (DMFT), where neurons asymptotically decouple and the macroscopic dynamics of the network are governed by population averages over the neurons in each hidden layer. This theory computes the dynamics of the learned representations of data in each hidden layer of the network throughout training as well as the dynamics of the network output predictions. Asymptotic corrections to this mean field limit will be computed from fluctuations around the DMFT saddle point. The mean field limit will then be stressed tested on several realistic networks such as convolutional networks and transformers on realistic computer vision and language modeling datasets. We next investigate the infinite depth limit of residual neural networks, where each hidden layer is a trainable perturbation of the identity map. When the residual branches are scaled correctly, these models admit infinite width and depth limits which are computable with a DMFT where intermediate layers approach a continuum limit described by the solution to a set of stochastic integral equations. We empirically show that scaling the depth and width correctly enable hyperparameter transfer, where optimal hyperparameters (learning rates, batch sizes, momentum values, etc) in small width and small depth models are the same in large width and large depth models, reducing the need for hyperparamter tuning. We will extend these mathematical techniques to analyze several distinct infinite-parameter limits for transformer models.Next, I will describe simplified models of learning which enable a statistical analysis of generalization in a data-limited or parameter-limited regime. As in the previous section, we present both static and dynamic versions of these results. For data distributions and architectures that generate power law spectra for their limiting kernels, these theories provide predictions for the scaling laws with respect to the key computational and statistical resources: training time, model parameters, and total available data. I show that this setting provides a toy model of compute optimal scaling laws where model size and training time are traded off optimally. Lastly, we use similar mathematical techniques to analyze a toy model of hyperparameter transfer in randomly initialized deep linear networks.Lastly, we move beyond randomly initialized deep networks and attempt to address how structured neural representations, such as the cortical representations of external stimuli in real brains, encode an implicit learning bias. We start with a simple neural circuit trained with the delta rule, finding that the spectral decomposition of the population code controls which learning tasks can be learned in a sample-efficient manner. Next, we examine how different learning rules alter feature learning dynamics and inductive bias in multilayer neural networks, extending the DMFT for multilayer networks to other biologically plausible learning rules. Lastly, we illustrate how the geometry of neural codes which respect symmetries in the data generating process controls the manifold capacity of linear readouts.

English

ISBN: 9798280710436Subjects--Topical Terms:

2122814
Applied mathematics.
Subjects--Index Terms:

Deep learning

Learning in Large Neural Networks /
LDR:04935nmm a22004213i 4500 001 2433862
005 20250905091030.5
006 m o d
007 cr|nu||||||||
008 261011s2025 miu||||||m |||||||eng d
020 $a 9798280710436
035 $a (MiAaPQD)AAI32041386
035 $a AAI32041386
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Bordelon, Blake, $e author. $0 (orcid)0000-0003-0455-9445 $3 3823635
245 1 0 $a Learning in Large Neural Networks / $c Blake Bordelon.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2025
300 $a 1 electronic resource (1396 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-12, Section: B.
500 $a Advisors: Pehlevan, Cengiz Committee members: Lu, Yue; Brenner, Michael; Sompolinsky, Haim.
502 $b Ph.D. $c Harvard University $d 2025.
520 $a In this thesis, I will summarize my recent works on theoretical frameworks for learning, generalization, scaling limits, and scaling laws of large neural networks. In the first part of the thesis, we will examine the kinds of limits attained by training randomly initialized networks. The two limits of primary interest are the infinite width and infinite depth feature-learning limits.The large width limit will take the form of a dynamical mean field theory (DMFT), where neurons asymptotically decouple and the macroscopic dynamics of the network are governed by population averages over the neurons in each hidden layer. This theory computes the dynamics of the learned representations of data in each hidden layer of the network throughout training as well as the dynamics of the network output predictions. Asymptotic corrections to this mean field limit will be computed from fluctuations around the DMFT saddle point. The mean field limit will then be stressed tested on several realistic networks such as convolutional networks and transformers on realistic computer vision and language modeling datasets. We next investigate the infinite depth limit of residual neural networks, where each hidden layer is a trainable perturbation of the identity map. When the residual branches are scaled correctly, these models admit infinite width and depth limits which are computable with a DMFT where intermediate layers approach a continuum limit described by the solution to a set of stochastic integral equations. We empirically show that scaling the depth and width correctly enable hyperparameter transfer, where optimal hyperparameters (learning rates, batch sizes, momentum values, etc) in small width and small depth models are the same in large width and large depth models, reducing the need for hyperparamter tuning. We will extend these mathematical techniques to analyze several distinct infinite-parameter limits for transformer models.Next, I will describe simplified models of learning which enable a statistical analysis of generalization in a data-limited or parameter-limited regime. As in the previous section, we present both static and dynamic versions of these results. For data distributions and architectures that generate power law spectra for their limiting kernels, these theories provide predictions for the scaling laws with respect to the key computational and statistical resources: training time, model parameters, and total available data. I show that this setting provides a toy model of compute optimal scaling laws where model size and training time are traded off optimally. Lastly, we use similar mathematical techniques to analyze a toy model of hyperparameter transfer in randomly initialized deep linear networks.Lastly, we move beyond randomly initialized deep networks and attempt to address how structured neural representations, such as the cortical representations of external stimuli in real brains, encode an implicit learning bias. We start with a simple neural circuit trained with the delta rule, finding that the spectral decomposition of the population code controls which learning tasks can be learned in a sample-efficient manner. Next, we examine how different learning rules alter feature learning dynamics and inductive bias in multilayer neural networks, extending the DMFT for multilayer networks to other biologically plausible learning rules. Lastly, we illustrate how the geometry of neural codes which respect symmetries in the data generating process controls the manifold capacity of linear readouts.
546 $a English
590 $a School code: 0084
650 4 $a Applied mathematics. $3 2122814
650 4 $a Computer science. $3 523869
650 4 $a Statistical physics. $3 536281
653 $a Deep learning
653 $a Scaling laws
653 $a Large neural networks
653 $a Convolutional networks
690 $a 0364
690 $a 0984
690 $a 0217
710 2 $a Harvard University. $b Engineering and Applied Sciences - Applied Math. $e degree granting institution. $3 3823171
720 1 $a Pehlevan, Cengiz $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-12B.
790 $a 0084
791 $a Ph.D.
792 $a 2025
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=32041386