Data Mining using Mahout SIEL@IIIT, Hyderabad（数据挖掘使用Mahout SIEL@IIIT,海德拉巴）.pdf

Data Mining using Mahout Data Mining using Mahout Team No. 8 Pratibha Rani Prashant Sethia Manisha Verma What is Mahout? What is Mahout? Subproject of Apache Lucene ◦Goal: delivering scalable machine learning algorithm implementations ◦/mahout/ Version 0.1 released on 07 April 2009 includes 10 algorithm libraries ◦Details in published paper: /people/ang//paper s/nips06-mapreducemulticore.pdf Objective Objective Implement two Data Mining/Machine Learning algorithms ◦Convert the algorithm in MapReduce paradigm ◦Implement using Hadoop ◦Optimize computation take advantage of MapReduce paradigm Integrate them in Mahout Library ◦Make it available online. Implemented Algorithms Implemented Algorithms Classification of Multi Class data using Linear Discriminant Function (LDF) ◦Machine Learning method for classification ◦Computational cost increases as number of classes increase SPRINT ◦Decision tree based parallel classifier for Data Mining ◦ Requires parallelization of computations Decision Tree Example Decision Tree Example Attribute Lists Attribute Lists Algorithm Algorithm Algorithm (contd.) Algorithm (contd.) SPRINT: Introduction SPRINT: Introduction Carry out decision tree building process in parallel ◦ Frequent lookup of the central class list produces a lot of network communication in the parallel case ◦ Solution: Eliminate the class list Class labels distributed to each attribute list = Redundant data, but the memory-resident and network communication bottlenecks are removed Each node keeps its own set of a