“So I have a bad history with Cloud Dataflow. I like BigQuery and Flink a lot, but a lot of companies seem to be switching over to Cloud Dataflow. In my experience Cloud Dataflow's cost is really unpredictable and DataPrep's pipelines (on DataFlow) were far more inefficient compared to an equivalent DataProc Spark pipeline, Spark on Compute Engine, or jus…”
Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems
With the demand to process ever-growing data volumes, a variety of new data stream processing frameworks have been developed. Moving an implementation from one such system to another, e.g., for performance reasons, requires adapting existing applications to new interfaces. Apache Beam addresses these high substitution costs by providing an abstraction layer that enables executing programs on any of the supported streaming frameworks. In this paper, we present a novel benchmark architecture for comparing the performance impact of using Apache Beam on three streaming frameworks: Apache Spark Streaming, Apache Flink, and Apache Apex. We find significant performance penalties when using Apache Beam for application development in the surveyed systems. Overall, usage of Apache Beam for the examined streaming applications caused a high variance of query execution times with a slowdown of up to a factor of 58 compared to queries developed without the abstraction layer. All developed benchmark artifacts are publicly available to ensure reproducible results.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check this reference out: https://arxiv.org/pdf/1907.08302.pdf “ Our benchmark results show that Apache Beam has a noticeable impact on the performance of DSPSs in almost all cases. Programs developed using Apache Beam suffered from a slowdown of up to a factor of 58 in the worst case. At the same time, there is one scenario where the query developed usi…”
- Quantitative Impact Evaluation of an Abstraction Layer for Stream Processing (2019) ↗jxub · 2020-03-05 · 2 分