東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Scheduling and Codesigning Machine L...

Fu, Daniel Y.,

Linked to FindBook

Google Book

Amazon

博客來

Scheduling and Codesigning Machine Learning Models for GEMMs /

Record Type:	Electronic resources : Monograph/item
Title/Author:	Scheduling and Codesigning Machine Learning Models for GEMMs // Daniel Y Fu.
Author:	Fu, Daniel Y.,
Description:	1 electronic resource (194 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-12, Section: B.
Contained By:	Dissertations Abstracts International86-12B.
Subject:	Sparsity. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31974593
ISBN:	9798311951364

Scheduling and Codesigning Machine Learning Models for GEMMs /
Fu, Daniel Y.,

Scheduling and Codesigning Machine Learning Models for GEMMs /Daniel Y Fu. - 1 electronic resource (194 pages)

Source: Dissertations Abstracts International, Volume: 86-12, Section: B.

Large-scale training of machine learning (ML) models on hardware accelerators like GPUs has led to substantial progress in recent years. Of critical importance is achieving high FLOP utilization during training. However, efficiently scheduling a diverse range of machine learning operators onto the specialized compute cores present in these hardware accelerators remains a challenge. For example, tensor core units, which are specialized to only compute matrix-matrix multiplication (GEMMs), compose up to 94% of the available FLOPs on Nvidia H100 GPUs. This specialization presents a challenge for modern ML models, which often interleave GEMMs with non-linearities (e.g., exponentiation), reduction operations (e.g., row-wise average), or non-GEMM operations (e.g., FFT).This thesis investigates how to schedule and codesign core machine learning operations onto specialized GEMM units, with a focus on Nvidia tensor cores (though the findings have been generalized to other hardware accelerators). We present two case studies on scheduling two core ML operations, attention and state space models (SSMs), followed by a study on codesigning a new architecture, Monarch Mixer, to utilize tensor cores from the ground-up. First, we focus on scheduling the attention operation, the core primitive at the heart of the Transformer architecture. We find that a simple algebraic reformulation of the softmax operation (combination of exponentiation and row-wise reduction) allows a tiling-based algorithm to achieve high FLOP utilization onto tensor cores. Second, we focus on SSMs, a promising next-generation architecture with better asymptotic complexity than Transformers. SSMs rely on a non-GEMM operation at their core, an FFT convolution. We show that an alternate algebraic formulation of the FFT, called a Monarch decomposition, allows the FFT convolution to be rewritten as a series of tensor core operations. Finally, we conduct a study on codesigning a new architecture, Monarch Mixer, to utilize tensor cores from the ground-up. We show that a generalized Monarch decomposition allows for a high-quality, asymptotically-efficient, and GEMM-friendly ML architecture. Overall, this thesis demonstrates that algebraic reformulation of ML operations is a powerful tool for scheduling diverse ML operations onto GEMMs.

English

ISBN: 9798311951364Subjects--Topical Terms:

3680690
Sparsity.

Scheduling and Codesigning Machine Learning Models for GEMMs /
LDR:03641nmm a22003853i 4500 001 2438303
005 20251210114004.5
006 m o d
007 cr|nu||||||||
008 261011s2025 miu||||||m |||||||eng d
020 $a 9798311951364
035 $a (MiAaPQD)AAI31974593
035 $a (MiAaPQD)Stanfordcs793yc4618
035 $a AAI31974593
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Fu, Daniel Y., $e author. $3 3829561
245 1 0 $a Scheduling and Codesigning Machine Learning Models for GEMMs / $c Daniel Y Fu.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2025
300 $a 1 electronic resource (194 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-12, Section: B.
500 $a Advisors: Fatahalian, Kayvon; Ré, Chris Committee members: Mirhoseini, Azalia.
502 $b Ph.D. $c Stanford University $d 2025.
520 $a Large-scale training of machine learning (ML) models on hardware accelerators like GPUs has led to substantial progress in recent years. Of critical importance is achieving high FLOP utilization during training. However, efficiently scheduling a diverse range of machine learning operators onto the specialized compute cores present in these hardware accelerators remains a challenge. For example, tensor core units, which are specialized to only compute matrix-matrix multiplication (GEMMs), compose up to 94% of the available FLOPs on Nvidia H100 GPUs. This specialization presents a challenge for modern ML models, which often interleave GEMMs with non-linearities (e.g., exponentiation), reduction operations (e.g., row-wise average), or non-GEMM operations (e.g., FFT).This thesis investigates how to schedule and codesign core machine learning operations onto specialized GEMM units, with a focus on Nvidia tensor cores (though the findings have been generalized to other hardware accelerators). We present two case studies on scheduling two core ML operations, attention and state space models (SSMs), followed by a study on codesigning a new architecture, Monarch Mixer, to utilize tensor cores from the ground-up. First, we focus on scheduling the attention operation, the core primitive at the heart of the Transformer architecture. We find that a simple algebraic reformulation of the softmax operation (combination of exponentiation and row-wise reduction) allows a tiling-based algorithm to achieve high FLOP utilization onto tensor cores. Second, we focus on SSMs, a promising next-generation architecture with better asymptotic complexity than Transformers. SSMs rely on a non-GEMM operation at their core, an FFT convolution. We show that an alternate algebraic formulation of the FFT, called a Monarch decomposition, allows the FFT convolution to be rewritten as a series of tensor core operations. Finally, we conduct a study on codesigning a new architecture, Monarch Mixer, to utilize tensor cores from the ground-up. We show that a generalized Monarch decomposition allows for a high-quality, asymptotically-efficient, and GEMM-friendly ML architecture. Overall, this thesis demonstrates that algebraic reformulation of ML operations is a powerful tool for scheduling diverse ML operations onto GEMMs.
546 $a English
590 $a School code: 0212
650 4 $a Sparsity. $3 3680690
650 4 $a Decomposition. $3 3561186
650 4 $a Benchmarks. $3 3548256
650 4 $a Computer science. $3 523869
690 $a 0984
690 $a 0800
710 2 $a Stanford University. $e degree granting institution. $3 3765820
720 1 $a Fatahalian, Kayvon $e degree supervisor.
720 1 $a Ré, Chris $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-12B.
790 $a 0212
791 $a Ph.D.
792 $a 2025
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31974593