《AWS_Amazon_EMR_Best_Practices》.pdf

Amazon Web Services – Best Practices for Amazon EMR August 2013 Best Practices for Amazon EMR Parviz Deyhim August 2013 (Please consult /whitepapers/ for the latest version of this paper) Page 1 of 38 Amazon Web Services – Best Practices for Amazon EMR August 2013 Table of Contents Abstract 3 Introduction 3 Moving Data to AWS4 Scenario 1: Moving Large Amounts of Data from HDFS (Data Center) to Amazon S3 4 Using S3DistCp 4 Using DistCp 5 Scenario 2: Moving Large Amounts of Data from Local Disk (non-HDFS) to Amazon S3 6 Using the Jets3t Java Library 6 Using GNU Parallel 7 Using Aspera Direct-to-S3 7 Using AWS Import/Export 7 Using AWS Direct Connect 8 Scenario 3: Moving Large Amounts of Data from Amazon S3 to HDFS 10 Using S3DistCp 10 Using DistCp 10 Data Collection 10 Using Apache Flume 11 Using Fluentd 11 Data Aggregation 12 Data Aggregation with Apache Flume 12 Data Aggregation Best Practices 13 Best Practice 1: Aggregated Data Size 14 Best Practice 2: Controlling Data Aggregation Size 14 Best Practice 3: Data Compression Algorithms 15 Best Practice 4: Data Partitioning 17 Processing Data with Amazon EMR 18 Picking the Right Instance Size 18 Picking the Right Number of Instances for Your Job 19 Estimating the Number of Mappers Your Job Requires 20 Amazon EMR Cluster Types 22 Transient Amazon EMR Clusters 20 Persistent Amazon EMR Clusters 22 Common Amazon EMR Architectures 23 Pattern