|
| | | | |
| | | |
| Stat |
Members: 3665
Articles: 2'599'751
Articles rated: 2609
25 January 2025 |
|
| | | | |
|
Article overview
| |
|
|
Co-Tuning of Cloud Infrastructure and Distributed Data Processing Platforms | | Isuru Dharmadasa
; Faheem Ullah
; | | Date: |
1 Sep 2023 | | Abstract: | Distributed Data Processing Platforms (e.g., Hadoop, Spark, and Flink) are
widely used to store and process data in a cloud environment. These platforms
distribute the storage and processing of data among the computing nodes of a
cloud. The efficient use of these platforms requires users to (i) configure the
cloud i.e., determine the number and type of computing nodes, and (ii) tune the
configuration parameters (e.g., data replication factor) of the platform.
However, both these tasks require in-depth knowledge of the cloud
infrastructure and distributed data processing platforms. Therefore, in this
paper, we first study the relationship between the configuration of the cloud
and the configuration of distributed data processing platforms to determine how
cloud configuration impacts platform configuration. After understanding the
impacts, we propose a co-tuning approach for recommending optimal
co-configuration of cloud and distributed data processing platforms. The
proposed approach utilizes machine learning and optimization techniques to
maximize the performance of the distributed data processing system deployed on
the cloud. We evaluated our approach for Hadoop, Spark, and Flink in a cluster
deployed on the OpenStack cloud. We used three benchmarking workloads
(WordCount, Sort, and K-means) in our evaluation. Our results reveal that, in
comparison to default settings, our co-tuning approach reduces execution time
by 17.5% and $ cost by 14.9% solely via configuration tuning. | | Source: | arXiv, 2309.00269 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
|
| |
|
|
|
REGISTER