programmingyournetworkatrun-timebigdataapplications-.PDF

Programming Your Network at Run-time for Big Data Applications Guohui Wang , T. S. Eugene Ng , Anees Shaikh IBM T.J. Watson Research Center, Rice University ABSTRACT While these earlier efforts have met with varying degrees of suc- Recent advances of software defined networking and optical switch- cess and adoption, the recent emergence of the software-defined ing technology make it possible to program the network stack all networking paradigm has created renewed interest in tailoring the the way from physical topology to flow level traffic control. In this network to better meet the needs of applications. By providing paper, we leverage the combination of SDN controller with optical a well-defined programming interface to the network (e.g., Open- switching to explore the tight integration of application and net- Flow), SDN provides an opportunity for more dynamic and flexi- work control. We particularly study the run-time network configu- ble interaction with the network. Despite this promising vision, the ration for big data applications to jointly optimize application per- ability of SDN to effectively configure or optimize the network to formance and network utilization. We use Hadoop as an example improve application performance and availability is still nascent. to discuss the integrated network control architecture, job schedul- Recently, two concomitant trends in data center applications and ing, topology and routing configuration mechanisms for Ha