interface design for spark community by xinreynoldxin为社区做界面设计.pdfVIP

Interfaces3

ReynoldXin

Aug22,2014@DatabricksRetreat

RepurposedJan27,2015forSparkcommunity

Spark’stwoimprovements

overHadoopMR

•Performance:“100X”fasterthanHadoopMR

•Programmingmodel:easiertouse

HadoopMRSpark

publicstaticclassWordCountMapClastendsMapReduceBaseimportorg.apache.spark.SparkContext

implementsMapperLongWritable,TeText,IntWritable{importorg.apache.spark.SparkContext._

valspark=newSparkContext(master,appName,[sparkHome],[jars])

valfile=spark.textFile(hdfs://...)

privatefinalstaticIntWritableone=newIntWritable(1);

valcounts=file.flatMap(line=line.split())

privateTextword=newText();.map(word=(word,1))

.reduceByKey(_+_)

publicvoidmap(LongWritablekey,Textvalue,counts.saveAsTextFile(hdfs://...)

OutputCollectorText,IntWritableoutput,

Reporterreporter)throwsIOException{

Stringline=value.toString();

StringTokenizeritr=newStringTokenizer(line);

while(itr.hasMoreTokens()){

word.set(itr.nextToken());

output.collect(word,one);

}“Ithasbeentheeasiest

}

}

publicstaticclassWorkdCountR