下一代分布式计算模型.pptx

© Hortonworks Inc. 2013 Page 1 YARN Apache Hadoop Next Generation Compute Platform Bikas Saha @bikassaha © Hortonworks Inc. 2013 - Confidential Apache Hadoop YARN • Apache Hadoop –De facto Big Data open source platform –Running for about 5 years in production at hundreds of companies like Yahoo, Ebay and Facebook • Hadoop 2 –Significant improvements in HDFS distributed storage layer. High Availability, NFS, Snapshots –YARN – next generation compute framework for Hadoop designed from the ground up based on experience gained from Hadoop 1 –YARN running in production at Yahoo for about a year –YARN awarded Best Paper at SOCC 2013 Page 2 © Hortonworks Inc. 2013 - Confidential Single App BATCH HDFS Single App INTERACTIVE Single App BATCH HDFS 1st Generation Hadoop: Batch Focus HADOOP 1.0 Built for Web-Scale Batch Apps All other usage patterns MUST leverage same infrastructure Forces Creation of Silos to Manage Mixed Workloads Single App BATCH HDFS Single App ONLINE Page 3 © Hortonworks Inc. 2013 - Confidential Hadoop 1 Architecture JobTracker Manage Cluster Resources Job Scheduling TaskTracker Per-node agent Manage Tasks Page 4 © Hortonworks Inc. 2013 - Confidential Hadoop 1 Limitations Lacks Support for Alternate Paradigms and Services Force everything needs to look like Map Reduce Iterative applications in MapReduce are 10x slower Scalability Max Cluster size ~5,000 nodes Max concurrent tasks ~40,000 Availability Failure Kills Queued Running Jobs Hard partition of resources into map and reduce slots Non-optimal Resource Utilization Page 5 Our Vision: Hadoop as Next-Gen Platform MapReduce (cluster resource management data processing) HDFS (redundant,reliable storage) © Hortonworks Inc. 2013 - Confidential YARN (cluster resource management) HDFS2 (redundant,highly-available reliable storage) MapReduce (data processing) Others Single Use System Batch Apps HADOOP 1.0 Multi Purpose Platform Batch, Interactive, Onli