MIS5208.docx

MIS 5208 Ed Ferrara, MSIA, CISSP eferrara@ Week 9: Big Data Splunk Agenda Chapter 1 Introduction / Splunk Big Data What is Big Data? Alternate Data Processing Techniques Machine Data What is Splunk? Chapter 2 Variety of Data Dealing with Data File Directories What is Big Data? The Three Vs Big Data are: High volume High velocity High variety Information assets that require new forms of processing to enable: Enhanced decision making Insight discovery Process optimization Volume – Data measured in petabytes Highway sensors Data processing logs Amazon purchase data Velocity – Speed of data generation and frequency of delivery Variety – Difference in the number of data types BIG DATA Facebook had more than 1B users with more than 618M active on a daily basis LinkedIn had more than 200M members – with the service adding 2 new members every second Instagram members upload 40M photos per day Twitter has 500M users – with the service adding 150K per day Wordpress has more than 40M new posts per day Pandora music streaming service has more than 13,700 years of music Etc. Splunk and the Kill Chain There are four classes of data that security teams need to leverage for a complete view: log data binary data (flow and PCAP) threat intelligence data and contextual data. If any of these data types are missing, there’s a higher risk that an attack will go unnoticed. These data types are the building blocks for knowing what’s normal and what’s not in your environment. This single question lies at the intersection of both system availability (IT operations and application) and security use cases. Splunk and the Kill Chain Effective data-driven security decisions require: Tens of terabytes of data per day without normalization Access data anywhere in the environment, including: Traditional security data sources Personnel time management systems HR databases Industrial control systems Hadoop data stores and custom enterprise applications that run the business Delivers fast tim