基于hadoop的网络数据分析.pptx

Hadoop-based Internet Traffic Analysis2012-03-14Youngseok Leemailto:lee@cnu.ac.krlee@cnu.ac.krChungnam National UniversityGartner: 10 Key IT Trends for 2012Data ExplosionDigital universe2005: 130 ExaByte2 time per two years90% of the data in the world today was created within the past two yearsMotivationCheap storageMobile computingBig DataBig data data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed timeExampleWalmart: 1 million customer transactions every hour2.5 PB of dataFacebook: 40 billion photos from users2.5 PB of user data + 15 TB/day (4/2009) eBay has 6.5 PB of user data + 50 TB/day (5/2009)Predicting Flu Trends with QueriesGinsberg, J., et al. (2009). Detecting influenza epidemics using search engine query data. Nature, 457, 1012-1014.Cutting Edge of Distributed/Parallel ComputingDistributed: MapReduce1 PB sorting by Google2008: 6 hours and 2 minutes on 4,000 computers 2011: 33 minutes on 8000 computersParallel: multi-core CPU, GPU, FPGA40Gbps IP forwarding capabilityS. Han et al., “PacketShader: A GPU-Accelerated Software Router“ ACM SIGCOMM, 2010Open-source framework for running applications on large clusters built of commodity hardwareImplementation of MapReduce and HDFSMapReduce : computational paradigmHDFS : distributed file systemMapReduceMapReduce is a programming model for large data setFirst suggested by GoogleJ. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Cluster,” OSDI, 2004 [8]User only specify a map and a reduce functionDistributed grepCount of URL access frequencyReverse web-link graphTerm-vector per hostInverted indexDistributed sortData from Internet MeasurementWebweb doc, blog, twitterIP packetsraw, header, dnsFlowCisco NetFlow v5/v9LogServer, switch, router, sensor, machine, appliancesManagement dataSNMP MIBRouting tableBGP update, tableActive probingping, traceroute, skitterHow Large ?4.5 TBType Packets4.6TB for 1 hr in 10