東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Towards a Cross-Layer Coupled Design...

Ye, Qianwen,

Linked to FindBook

Google Book

Amazon

博客來

Towards a Cross-Layer Coupled Design Framework for Big Data Workflows /

Record Type:	Electronic resources : Monograph/item
Title/Author:	Towards a Cross-Layer Coupled Design Framework for Big Data Workflows // Qianwen Ye.
Author:	Ye, Qianwen,
Description:	1 electronic resource (134 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-05, Section: B.
Contained By:	Dissertations Abstracts International86-05B.
Subject:	Computer science. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28969092
ISBN:	9798342726443

Towards a Cross-Layer Coupled Design Framework for Big Data Workflows /
Ye, Qianwen,

Towards a Cross-Layer Coupled Design Framework for Big Data Workflows /Qianwen Ye. - 1 electronic resource (134 pages)

Source: Dissertations Abstracts International, Volume: 86-05, Section: B.

The processing and analysis of big data increasingly rely on workflow technologies for knowledge discovery and scientific innovation. The execution of such workflows goes far beyond the capability and capacity of single computers and is now commonly supported on reliable and scalable data storage and analysis platforms in distributed environments, such as the Hadoop ecosystem. Workflow performance largely depends on how big data systems are configured and used. For example, the makespan of a big data workflow is affected by multiple layers of big data systems, including the parallel computing engine it runs on, the resource manager that orchestrates various types of computing resources, the distributed file system that partitions and stores data blocks, the network infrastructure that connects all computing nodes, as well as the parameter settings of each layer. Furthermore, there exist significant compound effects on workflow performance between different layers that oftentimes exhibit complex interactive behaviors. In this dissertation, a codesign framework for big data workflows is proposed, where the technology stack of big data systems is divided into multiple layers including application workflow, computing platform, resource management, data storage, and network infrastructure. In the network layer, a first-generation quantum repeater based on entanglement swapping is designed to support inter- or intra-module big data transfer. An emulator is implemented to test and evaluate the efficiency of a repeater hop using Mininet, and subsequently, the effects of network-related parameters are studied. In the resource management layer, a storage-aware scheduling algorithm is designed, which considers two layers of data movement to optimize workflow makespan. In the computing engine layer, the coupling effects of parameters are investigated and a stochastic approximation-based parameter tuning algorithm is designed to optimize the performance of Spark streaming applications. Lastly, a profiling-based cross-layer coupled design framework is proposed to determine the best parameter setting and identify the most suitable technique for each technology layer to optimize overall workflow performance. To tackle the large parameter space, two approaches are designed to reduce the number of experiments required for profiling: i) identify a subset of critical parameters with the most significant influence through information theory-based feature selection; and ii) minimize the search process within the value range of each critical parameter using stochastic approximation. The codesign framework together with the performance optimization techniques proposed in this dissertation is generic and hence is applicable to other big data systems to help users determine the most effective system configuration.

English

ISBN: 9798342726443Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Big data workflow

Towards a Cross-Layer Coupled Design Framework for Big Data Workflows /
LDR:04355nmm a22004453i 4500 001 2437618
005 20251124054947.5
006 m o d
007 cr|nu||||||||
008 261011s2022 miu||||||m |||||||eng d
020 $a 9798342726443
035 $a (MiAaPQD)AAI28969092
035 $a AAI28969092
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Ye, Qianwen, $e author. $3 3828669
245 1 0 $a Towards a Cross-Layer Coupled Design Framework for Big Data Workflows / $c Qianwen Ye.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2022
300 $a 1 electronic resource (134 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-05, Section: B.
500 $a Advisors: Wu, Chase Committee members: Schieber, Baruch M.; Li, Jing; Xu, Pan; Yu, Dantong; Katramatos, Dimitrios.
502 $b Ph.D. $c New Jersey Institute of Technology $d 2022.
520 $a The processing and analysis of big data increasingly rely on workflow technologies for knowledge discovery and scientific innovation. The execution of such workflows goes far beyond the capability and capacity of single computers and is now commonly supported on reliable and scalable data storage and analysis platforms in distributed environments, such as the Hadoop ecosystem. Workflow performance largely depends on how big data systems are configured and used. For example, the makespan of a big data workflow is affected by multiple layers of big data systems, including the parallel computing engine it runs on, the resource manager that orchestrates various types of computing resources, the distributed file system that partitions and stores data blocks, the network infrastructure that connects all computing nodes, as well as the parameter settings of each layer. Furthermore, there exist significant compound effects on workflow performance between different layers that oftentimes exhibit complex interactive behaviors. In this dissertation, a codesign framework for big data workflows is proposed, where the technology stack of big data systems is divided into multiple layers including application workflow, computing platform, resource management, data storage, and network infrastructure. In the network layer, a first-generation quantum repeater based on entanglement swapping is designed to support inter- or intra-module big data transfer. An emulator is implemented to test and evaluate the efficiency of a repeater hop using Mininet, and subsequently, the effects of network-related parameters are studied. In the resource management layer, a storage-aware scheduling algorithm is designed, which considers two layers of data movement to optimize workflow makespan. In the computing engine layer, the coupling effects of parameters are investigated and a stochastic approximation-based parameter tuning algorithm is designed to optimize the performance of Spark streaming applications. Lastly, a profiling-based cross-layer coupled design framework is proposed to determine the best parameter setting and identify the most suitable technique for each technology layer to optimize overall workflow performance. To tackle the large parameter space, two approaches are designed to reduce the number of experiments required for profiling: i) identify a subset of critical parameters with the most significant influence through information theory-based feature selection; and ii) minimize the search process within the value range of each critical parameter using stochastic approximation. The codesign framework together with the performance optimization techniques proposed in this dissertation is generic and hence is applicable to other big data systems to help users determine the most effective system configuration.
546 $a English
590 $a School code: 0152
650 4 $a Computer science. $3 523869
650 4 $a Computer engineering. $3 621879
650 4 $a Quantum physics. $3 726746
653 $a Big data workflow
653 $a Cross-layer optimization
653 $a Performance optimization
653 $a Quantum network
653 $a Scheduling algorithm
653 $a Searching algorithm
690 $a 0984
690 $a 0464
690 $a 0599
710 2 $a New Jersey Institute of Technology. $b Department of Computer Science. $e degree granting institution. $3 3828649
720 1 $a Wu, Chase $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-05B.
790 $a 0152
791 $a Ph.D.
792 $a 2022
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28969092