1622 IEEE COMMUNICATIONS SURVEYS TUTORIALS.VOL.16.NO.3.THIRD QUARTER 2014 scalability of the network.In the following,we address some Applications such scalability concerns and go over some proposals that aim on overcoming these challenges.We leave a more detailed Network OS discussion on the application layer and the implementation of services and policy enforcement to Section VI-C. Decoupled 1)Control Scalability:An initial concern that arises when Control Logic offloading control from the switching hardware is the scalabil- ity and performance of the network controller(s).The original Secure Ethane [20]controller,hosted on a commodity desktop ma- 可 Channel chine,was tested to handle up to 11,000 new flow requests per second and responded within 1.5 milliseconds.A more recent Abstraction Layer study [39]of several OpenFlow controller implementations (NOX-MT,Maestro,Beacon),conducted on a larger emulated network with 100,000 endpoints and up to 256 switches,found that all were able to handle at least 50,000 new flow requests Flow Table per second in each of the tested scenarios.On an eight- core machine,the multi-threaded NOX-MT implementation handled 1.6 million new flow requests per second with an SWITCH average response time of 2 milliseconds.As the results show, a single controller is able to handle a surprising number of new flow requests,and should be able to manage all but the largest Fig.3.The separated control logic can be viewed as a network operating networks.Furthermore,new controllers under development system,upon which applications can be built to "program"the network. such as McNettle [40]target powerful multicore servers and are being designed to scale up to large data center workloads C.The Controller (around 20 million flows requests per second and up to 5000 The decoupled system has been compared to an operating switches).Nonetheless,multiple controllers may be used to system [17],in which the controller provides a programmatic reduce latency or increase fault tolerance. interface to the network.That can be used to implement A related concern is the controller placement problem [41]. management tasks and offer new functionalities.A layered which attempts to determine both the optimal number of view of this model is illustrated in Figure 3.This abstraction controllers and their location within the network topology, assumes the control is centralized and applications are written often choosing between optimizing for average and worst as if the network is a single system.It enables the SDN case latency.The latency of the link used for communication model to be applied over a wide range of applications and between controller and switch is of great importance when heterogeneous network technologies and physical media such dimensioning a network or evaluating its performance [34]. as wireless (e.g.802.11 and 802.16),wired (e.g.Ethernet)and That was one of the main motivations behind the work in [42] optical networks. which evaluated how the controller and the network perform As a practical example of the layering abstraction accessi-with bandwidth and latency issues on the control link.This ble through open application programming interfaces(APIs),work concludes that bandwidth in the control link arbitrates Figure 4 illustrates the architecture of an SDN controller how many flows can be processed by the controller,as well based on the OpenFlow protocol.This specific controller is as the loss rate when under saturation conditions.The switch- a fork of the Beacon controller [22]called Floodlight [38]. to-control latency on the other hand,has a major impact on In this figure it is possible to observe the separation between the overall behavior of the network,as each switch cannot the controller and the application layers.Applications can be forward data until it receives the message from the controller written in Java and can interact with the built-in controller that inserts the appropriate rules in the flow table.This interval modules via a JAVA APl.Other applications can be written in can grow with the link latency and impact dramatically the different languages and interact with the controller modules performance of network applications. via the REST API.This particular example of an SDN Also,control modeling greatly impacts the network scal- controller allows the implementation of built-in modules that ability.Some important scalability issues are presented can communicate with their implementation of the OpenFlow in [43],along with a discussion about scalability trade-offs controller (i.e.OpenFlow Services).The controller,on the in software-defined network design. other hand,can communicate with the forwarding devices via 2)Control models:In the following,we go over some of the OpenFlow protocol through the abstraction layer present these SDN design options and discuss different methods of at the forwarding hardware,illustrated in Figure 3. controlling a software-defined network,many of which are While the aforementioned layering abstractions accessible interrelated: via open APIs allow the simplification of policy enforce- Centralized vs.Distributed ment and management tasks,the bindings must be closely Although protocols such as OpenFlow specify that a maintained between the control and the network forwarding switch is controlled by a controller and therefore ap- elements.The choices made while implementing such layering pears to imply centralization,software-defined networks architectures can dramatically influence the performance and may have either a centralized or distributed control-
1622 IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 16, NO. 3, THIRD QUARTER 2014 Network OS Applications Secure Channel Decoupled Control Logic SWITCH Flow Table Abstraction Layer Fig. 3. The separated control logic can be viewed as a network operating system, upon which applications can be built to “program” the network. C. The Controller The decoupled system has been compared to an operating system [17], in which the controller provides a programmatic interface to the network. That can be used to implement management tasks and offer new functionalities. A layered view of this model is illustrated in Figure 3. This abstraction assumes the control is centralized and applications are written as if the network is a single system. It enables the SDN model to be applied over a wide range of applications and heterogeneous network technologies and physical media such as wireless (e.g. 802.11 and 802.16), wired (e.g. Ethernet) and optical networks. As a practical example of the layering abstraction accessible through open application programming interfaces (APIs), Figure 4 illustrates the architecture of an SDN controller based on the OpenFlow protocol. This specific controller is a fork of the Beacon controller [22] called Floodlight [38]. In this figure it is possible to observe the separation between the controller and the application layers. Applications can be written in Java and can interact with the built-in controller modules via a JAVA API. Other applications can be written in different languages and interact with the controller modules via the REST API. This particular example of an SDN controller allows the implementation of built-in modules that can communicate with their implementation of the OpenFlow controller (i.e. OpenFlow Services). The controller, on the other hand, can communicate with the forwarding devices via the OpenFlow protocol through the abstraction layer present at the forwarding hardware, illustrated in Figure 3. While the aforementioned layering abstractions accessible via open APIs allow the simplification of policy enforcement and management tasks, the bindings must be closely maintained between the control and the network forwarding elements. The choices made while implementing such layering architectures can dramatically influence the performance and scalability of the network. In the following, we address some such scalability concerns and go over some proposals that aim on overcoming these challenges. We leave a more detailed discussion on the application layer and the implementation of services and policy enforcement to Section VI-C. 1) Control Scalability: An initial concern that arises when offloading control from the switching hardware is the scalability and performance of the network controller(s). The original Ethane [20] controller, hosted on a commodity desktop machine, was tested to handle up to 11,000 new flow requests per second and responded within 1.5 milliseconds. A more recent study [39] of several OpenFlow controller implementations (NOX-MT, Maestro, Beacon), conducted on a larger emulated network with 100,000 endpoints and up to 256 switches, found that all were able to handle at least 50,000 new flow requests per second in each of the tested scenarios. On an eightcore machine, the multi-threaded NOX-MT implementation handled 1.6 million new flow requests per second with an average response time of 2 milliseconds. As the results show, a single controller is able to handle a surprising number of new flow requests, and should be able to manage all but the largest networks. Furthermore, new controllers under development such as McNettle [40] target powerful multicore servers and are being designed to scale up to large data center workloads (around 20 million flows requests per second and up to 5000 switches). Nonetheless, multiple controllers may be used to reduce latency or increase fault tolerance. A related concern is the controller placement problem [41], which attempts to determine both the optimal number of controllers and their location within the network topology, often choosing between optimizing for average and worst case latency. The latency of the link used for communication between controller and switch is of great importance when dimensioning a network or evaluating its performance [34]. That was one of the main motivations behind the work in [42] which evaluated how the controller and the network perform with bandwidth and latency issues on the control link. This work concludes that bandwidth in the control link arbitrates how many flows can be processed by the controller, as well as the loss rate when under saturation conditions. The switchto-control latency on the other hand, has a major impact on the overall behavior of the network, as each switch cannot forward data until it receives the message from the controller that inserts the appropriate rules in the flow table. This interval can grow with the link latency and impact dramatically the performance of network applications. Also, control modeling greatly impacts the network scalability. Some important scalability issues are presented in [43], along with a discussion about scalability trade-offs in software-defined network design. 2) Control models: In the following, we go over some of these SDN design options and discuss different methods of controlling a software-defined network, many of which are interrelated: • Centralized vs. Distributed Although protocols such as OpenFlow specify that a switch is controlled by a controller and therefore appears to imply centralization, software-defined networks may have either a centralized or distributed control-
NUNES et al:A SURVEY OF SOFTWARE-DEFINED NETWORKING:PAST.PRESENT.AND FUTURE OF PROGRAMMABLE NETWORKS 1623 Leaming PortDown OpenStack Switch Reconciliation Quantum Plugin Firewall VNF Hub Circuit Pusher JAVA API REST API Module Thread Packet Jython Unit Manager Web Ul Pool Streamer Server Tests Device Topology Link Flow Manager Manager/ Discovery Cache Storage Memory Routing Controller Counter Switches PerfMon Trace Memory Store OpenFlow Services Fig.4.The Floodlight architecture as an example of an OpenFlow controller. plane.Though controller-to-controller communication is sor [48].can be used to add a level of network virtualiza- not defined by OpenFlow,it is necessary for any type of tion to OpenFlow networks and allow multiple controllers distribution or redundancy in the control-plane. to simultaneously control overlapping sets of physical A physically centralized controller represents a single switches.Initially developed to allow experimental re- point of failure for the entire network;therefore,Open- search to be conducted on deployed networks alongside Flow allows the connection of multiple controllers to a production traffic,it also facilitates and demonstrates the switch,which would allow backup controllers to take ease of deploying new services in SDN environments. over in the event of a failure. A logically decentralized control plane would be needed Onix [44]and HyperFlow [45]take the idea further in an inter-network spanning multiple administrative do- by attempting to maintain a logically centralized but mains.Though the domains may not agree to centralized physically distributed control plane.This decreases the control,a certain level of sharing may be appropriate look-up overhead by enabling communication with local (e.g.,to ensure service level agreements are met for traffic controllers,while still allowing applications to be written flowing between domains). with a simplified central view of the network.The po- 。Control Granularity tential downside are trade-offs [46]related to consistency Traditionally,the basic unit of networking has been and staleness when distributing state throughout the con- the packet.Each packet contains address information trol plane,which has the potential to cause applications necessary for a network switch to make routing decisions. that believe they have an accurate view of the network However,most applications send data as a flow of many to act incorrectly. individual packets.A network that wishes to provide A hybrid approach,such as Kandoo [47],can utilize local QoS or service guarantees to certain applications may controllers for local applications and redirect to a global benefit from individual flow-based control.Control can controller for decisions that require centralized network be further abstracted to an aggregated flow-match,rather state.This reduces the load on the global controller by than individual flows.Flow aggregation may be based filtering the number of new flow requests,while also on source,destination,application,or any combination providing the data-path with faster responses for requests thereof. that can be handled by a local control application. In a software-defined network where network elements A software-defined network can also have some level of are controlled remotely,overhead is caused by traffic logical decentralization,with multiple logical controllers. between the data-plane and control-plane.As such,using An interesting type of proxy controller,called Flowvi- packet level granularity would incur additional delay as
NUNES et al.: A SURVEY OF SOFTWARE-DEFINED NETWORKING: PAST, PRESENT, AND FUTURE OF PROGRAMMABLE NETWORKS 1623 Module Manager Thread Pool Packet Streamer Jython Server Web UI Unit Tests Device Manager Topology Manager/ Routing Link Discovery Flow Cache Storage Memory Switches Controller Memory PerfMon Trace Counter Store Firewall Hub Learning Switch PortDown Reconciliation VNF Circuit Pusher OpenStack Quantum Plugin OpenFlow Services JAVA API Fig. 4. The Floodlight architecture as an example of an OpenFlow controller. plane. Though controller-to-controller communication is not defined by OpenFlow, it is necessary for any type of distribution or redundancy in the control-plane. A physically centralized controller represents a single point of failure for the entire network; therefore, OpenFlow allows the connection of multiple controllers to a switch, which would allow backup controllers to take over in the event of a failure. Onix [44] and HyperFlow [45] take the idea further by attempting to maintain a logically centralized but physically distributed control plane. This decreases the look-up overhead by enabling communication with local controllers, while still allowing applications to be written with a simplified central view of the network. The potential downside are trade-offs [46] related to consistency and staleness when distributing state throughout the control plane, which has the potential to cause applications that believe they have an accurate view of the network to act incorrectly. A hybrid approach, such as Kandoo [47], can utilize local controllers for local applications and redirect to a global controller for decisions that require centralized network state. This reduces the load on the global controller by filtering the number of new flow requests, while also providing the data-path with faster responses for requests that can be handled by a local control application. A software-defined network can also have some level of logical decentralization, with multiple logical controllers. An interesting type of proxy controller, called Flowvisor [48], can be used to add a level of network virtualization to OpenFlow networks and allow multiple controllers to simultaneously control overlapping sets of physical switches. Initially developed to allow experimental research to be conducted on deployed networks alongside production traffic, it also facilitates and demonstrates the ease of deploying new services in SDN environments. A logically decentralized control plane would be needed in an inter-network spanning multiple administrative domains. Though the domains may not agree to centralized control, a certain level of sharing may be appropriate (e.g., to ensure service level agreements are met for traffic flowing between domains). • Control Granularity Traditionally, the basic unit of networking has been the packet. Each packet contains address information necessary for a network switch to make routing decisions. However, most applications send data as a flow of many individual packets. A network that wishes to provide QoS or service guarantees to certain applications may benefit from individual flow-based control. Control can be further abstracted to an aggregated flow-match, rather than individual flows. Flow aggregation may be based on source, destination, application, or any combination thereof. In a software-defined network where network elements are controlled remotely, overhead is caused by traffic between the data-plane and control-plane. As such, using packet level granularity would incur additional delay as