大数据数据挖掘培训讲义1：机器学习数据挖掘知识发现简介.ppt

COM 307: Machine Learning and Data Mining Machine Learning,Data Mining, andKnowledge Discovery: An Introduction Gregory Piatetsky-Shapiro KDnuggets Course Outline Machine Learning input, representation, decision trees Weka machine learning workbench Data Mining associations, deviation detection, clustering, visualization Case Studies targeted marketing, genomic microarrays Data Mining, Privacy and Security Final Project: Microarray Data Mining Competition Lesson Outline Introduction: Data Flood Data Mining Application Examples Data Mining Knowledge Discovery Data Mining Tasks Trends leading to Data Flood More data is generated: Bank, telecom, other business transactions ... Scientific data: astronomy, biology, etc Web, text, and e-commerce Big Data Examples Europes Very Long Baseline Interferometry (VLBI) has 16 telescopes, each of which produces 1 Gigabit/second of astronomical data over a 25-day observation session storage and analysis a big problem ATT handles billions of calls per day so much data, it cannot be all stored -- analysis has to be done “on the fly”, on streaming data Largest databases in 2003 Commercial databases: Winter Corp. 2003 Survey: France Telecom has largest decision-support DB, ~30TB; ATT ~ 26 TB Web Alexa internet archive: 7 years of data, 500 TB Google searches 4+ Billion pages, many hundreds TB IBM WebFountain, 160 TB (2003) Internet Archive (),~ 300 TB From terabytes to exabytes to … UC Berkeley 2003 estimate: 5 exabytes (5 million terabytes) of new data was created in 2002. /research/projects/how-much-info-2003/ US produces ~40% of new stored data worldwide 2006 estimate: 161 exabytes (IDC study) /tech/news/2007-03-05-data_N.htm 2010 projection: 988 exabytes Largest Databases in 2005 Winter Corp. 2005 Commercial Database Survey: Max Planck Inst. for Meteorology , 222 TB Yahoo ~ 100 TB (Largest Data Warehouse) ATT ~ 94 TB Data Growth Data Growth Rate Twice as much information was created in 2002 as in 1999 (~30% growt