東華大學圖書館 |

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks./
作者:	Abusalah, Bara.
出版者:	Ann Arbor : ProQuest Dissertations & Theses, : 2021,
面頁冊數:	104 p.
附註:	Source: Dissertations Abstracts International, Volume: 85-01, Section: B.
Contained By:	Dissertations Abstracts International85-01B.
標題:	End users. -
電子資源:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30504653
ISBN:	9798379829179

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.
Abusalah, Bara.

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks. - Ann Arbor : ProQuest Dissertations & Theses, 2021 - 104 p.

Source: Dissertations Abstracts International, Volume: 85-01, Section: B.

Thesis (Ph.D.)--Purdue University, 2021.

The examiner of cloud computing systems in the last few years observes that there is a trend of the emergence of new Big Data frameworks every single year. Since Hadoop was developed in 2007, new frameworks followed it such as Spark, Storm, Heron, Apex, Flink, Samza, Kafka ... etc. Each framework is developed in a certain way to target and achieve certain objectives better than other frameworks do. However, there are few common functionalities and aspects that are shared between these frameworks. One vital aspect all these frameworks strive to achieve is better reliability and faster recovery time in case of failures. Despite all the advances in making datacenters dependable, failures actually still happen. This is particularly onerous for long-running "big data" applications, where partial failures can lead to significant losses and lengthy recomputations. This is also crucial for streaming systems where events are processed and monitored online in real time, and any delay in data delivery will cause a major inconvenience to the users.Another observation is that some reliability implementations are redundant between different frameworks. Big data processing frameworks like Hadoop MapReduce include fault tolerance mechanisms, but these are commonly targeted at specific system/failure models, and are often redundant between frameworks. Encapsulating these implementations into one layer and making it shared between different applications will benefit more than one framework without the burden of re-implementing the same reliability approach in each single framework.These observations motivated us to solve the problem by presenting two systems: Guardian and Warden. Guardian is tailored towards batch processing big data systems while Warden is targeted towards stream processing systems. Both systems are robust, RMS based, generic, multi-framework, flexible, customizable, low overhead systems that allow their users to run their applications with individually configurable fault tolerance granularity and degree, with only minor changes to their implementation.Most reliability approaches carry out one rigid fault tolerance technique targeted towards one system at a time. It is more challenging to provide a reliability approach that is pluggable in multiple Big Data frameworks at a time and can achieve low overheads comparable with single targeted framework approaches, yet is flexible and customizable by its users to make it tailored towards their objectives. The genericity is attained by providing an interface that can be used in different applications from different frameworks in any part of the application code. The low overhead is achieved by providing faster application finish times with and without failures. The customizability is fulfilled by providing the users the options to choose between two fault tolerance guarantees (Crash Failures / Byzantine Failures) and, in case of streaming systems; it is combined with two delivery semantics (Exactly Once / At Most Once).In other words, this thesis proposes the paradigm of dependable resources: big data processing frameworks are typically built on top of resource management systems (RMSs), and proposing fault tolerance support at the level of such an RMS yields generic fault tolerance mechanisms, which can be provided with low overhead by leveraging constraints on resources.To the best of our knowledge, such approach was never tried on multiple big data batch processing and streaming frameworks before.We demonstrate the benefits of Guardian by evaluating some batch processing frameworks such as Hadoop, Tez, Spark and Pig on a prototype of Guardian running on AmazonEC2, improving completion time by around 68% in the presence of failures, while maintaining around 6% overhead. We've also built a prototype of Warden on the Flink and Samza (with Kafka) streaming frameworks. Our evaluations on Warden highlight the effectiveness of our approach in the presence of failures and without failures compared to other fault tolerance techniques (such as checkpointing).

ISBN: 9798379829179Subjects--Topical Terms:

3680535
End users.

Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.
LDR:05051nmm a2200313 4500 001 2399172
005 20240909100741.5
006 m o d
007 cr#unu||||||||
008 251215s2021 ||||||||||||||||| ||eng d
020 $a 9798379829179
035 $a (MiAaPQ)AAI30504653
035 $a (MiAaPQ)Purdue14477733
035 $a AAI30504653
040 $a MiAaPQ $c MiAaPQ
100 1 $a Abusalah, Bara. $3 3769141
245 1 0 $a Dependable Cloud Resources For Big-Data Batch Processing & Streaming Frameworks.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2021
300 $a 104 p.
500 $a Source: Dissertations Abstracts International, Volume: 85-01, Section: B.
500 $a Advisor: Ghafoor, Arif.
502 $a Thesis (Ph.D.)--Purdue University, 2021.
520 $a The examiner of cloud computing systems in the last few years observes that there is a trend of the emergence of new Big Data frameworks every single year. Since Hadoop was developed in 2007, new frameworks followed it such as Spark, Storm, Heron, Apex, Flink, Samza, Kafka ... etc. Each framework is developed in a certain way to target and achieve certain objectives better than other frameworks do. However, there are few common functionalities and aspects that are shared between these frameworks. One vital aspect all these frameworks strive to achieve is better reliability and faster recovery time in case of failures. Despite all the advances in making datacenters dependable, failures actually still happen. This is particularly onerous for long-running "big data" applications, where partial failures can lead to significant losses and lengthy recomputations. This is also crucial for streaming systems where events are processed and monitored online in real time, and any delay in data delivery will cause a major inconvenience to the users.Another observation is that some reliability implementations are redundant between different frameworks. Big data processing frameworks like Hadoop MapReduce include fault tolerance mechanisms, but these are commonly targeted at specific system/failure models, and are often redundant between frameworks. Encapsulating these implementations into one layer and making it shared between different applications will benefit more than one framework without the burden of re-implementing the same reliability approach in each single framework.These observations motivated us to solve the problem by presenting two systems: Guardian and Warden. Guardian is tailored towards batch processing big data systems while Warden is targeted towards stream processing systems. Both systems are robust, RMS based, generic, multi-framework, flexible, customizable, low overhead systems that allow their users to run their applications with individually configurable fault tolerance granularity and degree, with only minor changes to their implementation.Most reliability approaches carry out one rigid fault tolerance technique targeted towards one system at a time. It is more challenging to provide a reliability approach that is pluggable in multiple Big Data frameworks at a time and can achieve low overheads comparable with single targeted framework approaches, yet is flexible and customizable by its users to make it tailored towards their objectives. The genericity is attained by providing an interface that can be used in different applications from different frameworks in any part of the application code. The low overhead is achieved by providing faster application finish times with and without failures. The customizability is fulfilled by providing the users the options to choose between two fault tolerance guarantees (Crash Failures / Byzantine Failures) and, in case of streaming systems; it is combined with two delivery semantics (Exactly Once / At Most Once).In other words, this thesis proposes the paradigm of dependable resources: big data processing frameworks are typically built on top of resource management systems (RMSs), and proposing fault tolerance support at the level of such an RMS yields generic fault tolerance mechanisms, which can be provided with low overhead by leveraging constraints on resources.To the best of our knowledge, such approach was never tried on multiple big data batch processing and streaming frameworks before.We demonstrate the benefits of Guardian by evaluating some batch processing frameworks such as Hadoop, Tez, Spark and Pig on a prototype of Guardian running on AmazonEC2, improving completion time by around 68% in the presence of failures, while maintaining around 6% overhead. We've also built a prototype of Warden on the Flink and Samza (with Kafka) streaming frameworks. Our evaluations on Warden highlight the effectiveness of our approach in the presence of failures and without failures compared to other fault tolerance techniques (such as checkpointing).
590 $a School code: 0183.
650 4 $a End users. $3 3680535
650 4 $a Data processing. $3 680224
650 4 $a Failure. $3 3561225
650 4 $a Computer science. $3 523869
650 4 $a Batch processing. $3 3680543
650 4 $a Fault tolerance. $3 3561030
650 4 $a Cloud computing. $3 1016782
650 4 $a Real time. $3 3562675
690 $a 0984
710 2 $a Purdue University. $3 1017663
773 0 $t Dissertations Abstracts International $g 85-01B.
790 $a 0183
791 $a Ph.D.
792 $a 2021
793 $a English
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=30504653