mapreduce数据分析.ppt

作者2 Erik Paulson， University of Wisconsin 1 MapReduce and parallel DBMSs: friends or foes? 2 A comparison of approaches to large-scale data analysis 3 Clustera: an integrated computation and data management system 和第一作者一样，主要做Hadoop(Mapreduce)和并行数据库管理系统比较，用于大规模数据集分析。 作者3 Alexander Rasin ，Brown University 1 CORADD: correlation aware database designer for materialized views and indexes 2 MapReduce and parallel DBMSs: friends or foes? 3 HadoopDB: an architectural hybrid of MapReduce and DBMS technologies for analytical workloads 4 Correlation maps: a compressed access method for exploiting soft functional dependencies 5 A comparison of approaches to large-scale data analysis 6 H-store: a high-performance, distributed main memory transaction processing system 作者在本文的基础上，设计了HadoopDB系统，一个Mapreduce和并行数据库管理系统结合的系统。 摘要 目前有相当大的兴趣在基于MapReduce（MR）模式的大规模数据分析。虽然这个框架的基本控制流已经存在于并行SQL数据库管理系统超过20年，也有人称MR为必威体育精装版的计算模型。在本文中，我们描述和比较这两个模式。此外，我们评估两个系统的性能和开发复杂度。最后，我们定义一个包含任务集的基准运行于MR开源平台和两个并行数据库管理系统上。对于每个任务，我们在100台机子的集群上衡量每个系统的各个方面的并行性能。我们的研究结果揭示了一些有趣的取舍。虽然加载数据和调整并行数据库管理系统执行的过程比MR花费更多的时间，但是观察到的这些数据库管理系统性能显著地改善。我们推测巨大的性能差异的原因，并考虑将来的系统应该从这两种架构中吸取优势。 ABSTRACT：There is currently considerable enthusiasm around the MapReduce (MR) paradigm for large-scale data analysis. Although the basic control ?ow of this framework has existed in parallel SQL database management systems (DBMS) for over 20 years, some have called MR a dramatically new computing model. In this paper, we describe and compare both paradigms. Furthermore, we evaluate both kinds of systems in terms of performance and development complexity. To this end, we de?ne a benchmark consisting of a collection of tasks that we have run on an open source version of MR as well as on two parallel DBMSs. For each task, we measure each system’s performance for various degrees of parallelism on a cluster of 100 nodes. Our results reveal some interesting trade-offs. Although the process to load data into and tune the execution of parallel DBMSs took m