《Spark-Yarn-Cloudera》.pdf

Can’t We All Just Get Along? Spark and Resource Management on Hadoop Introducons •  Soware engineer at Cloudera •  MapReduce, YARN, Resource management •  Hadoop commier Introducon •  Spark as a first class data processing framework alongside MR and Impala •  Resource management •  What we have already •  What we need for the future Bringing Computaon to the Data •  Users want to •  ETL a dataset with Pig and MapReduce •  Fit a model to it with Spark •  Have BI tools query it with Impala •  Same set of machines that hold data must also host these frameworks Cluster Resource Management •  Hadoop brings generalized computaon to big data •  More processing frameworks •  MapReduce, Impala, Spark •  Some workloads are more important than others •  A cluster has finite resources •  Limited CPU, memory, disk and network bandwidth •  How do we make sure each workload gets the resources it deserves? How We See It Impala MapReduce Spark HDFS How They Want to See It Engineering - 50% Finance - 30% Marketing - 20% Spark Spark Spark MR MR MR Impala Impala Impala HDFS Central Resource Management Impala MapReduce Spark YARN HDFS YARN •  Resource manager and scheduler for Hadoop •  “Container” is a process scheduled on the cluster with a resource allocaon (amount MB, # cores) •  E