MapReduce培训课件.ppt

* * * Performance Tests run on cluster of 1800 machines: 4 GB of memory Dual-processor 2 GHz Xeons with Hyperthreading Dual 160 GB IDE disks Gigabit Ethernet per machine Bisection bandwidth approximately 100 Gbps Two benchmarks: MR_Grep Scan 1010 100-byte records to extract records matching a rare pattern (92K matching records) MR_Sort Sort 1010 100-byte records (modeled after TeraSort benchmark) MR_Grep Locality optimization helps: 1800 machines read 1 TB of data at peak of ~31 GB/s Without this, rack switches would limit to 10 GB/s Startup overhead is significant for short jobs MR_Sort Backup tasks reduce job completion time significantly System deals well with failures Normal No Backup Tasks 200 processes killed More and more MapReduce MapReduce Programs In Google Source Tree Example uses: distributed grep distributed sort web link-graph reversal term-vector per host web access log stats inverted index construction document clustering machine learning statistical machine translation MapReduce Conclusions MapReduce has proven to be a useful abstraction Greatly simplifies large-scale computations at Google Functional programming paradigm can be applied to large-scale applications Fun to use: focus on problem, let library deal w/ messy details Further Improvement “Improving MapReduce Performance in Heterogeneous Environments” in OSDI’08 * * * * * * * * * * * * * * * * * * * * * * * * * * MapReduce课件 Outline MapReduce overview Discussion Questions MapReduce Motivation 200+ processors 200+ terabyte database 1010 total clock cycles 0.1 second response time 5￠ average advertising revenue From: /~bryant/presentations/DISC-FCRC07.ppt Motivation: Large Scale Data Processing Want to process lots of data ( 1 TB) Want to parallelize across hundreds/thousands of CPUs … Want to make this easy Google Earth uses 70.5 TB: 70 TB for the raw imagery and 500 GB for the index data. From: /2006/09/how-much-data-does-google-store.html MapReduce Automatic paralleli