In today's data centers, local area networking (LAN), storage area networking (SAN) and inter-processor communications (IPC) use different networking technologies, such as Ethernet, Fibre Channel and InfiniBand, respectively. As a result, IT management must contend with two or three different networks, network adapter and switch technologies. This fragmented network infrastructure makes optimizing server performance, scaling server capacity and managing space and power costs difficult.

InfiniBand is typically seen as a technology that is useful only in high-performance computing (HPC) environments, but many of the same attributes that make it the choice of HPC network architects can also simplify data center infrastructure, while boosting performance and flexibility and reducing costs.

InfiniBand meets the service levels needed by LAN, SAN and IPC traffic with a single wire. Because InfiniBand delivers higher levels of performance and services than other interconnect technologies, a single InfiniBand adapter can replace multiple Fibre Channel and Ethernet adapter cards from each server, resulting in immediate cost, power and infrastructure equipment savings.

IT managers can use modular InfiniBand-to-Ethernet or InfiniBand-to-Fibre Channel gateways for connectivity to Ethernet-based LAN/WAN and Fibre Channel-based SAN networks, enabling cost-effective SAN and LAN capacity provisioning and scaling, and an end-to-end experience with Ethernet and Fibre Channel technologies.

InfiniBand protocol software, available in all popular operating systems, delivers industry-standard application interfaces like IP, sockets, small computer systems interface (SCSI), Internet small computer system interface (iSCSI) and network file system (NFS), which make application deployment transparent to the underlying data center fabric.

As a unified I/O technology, InfiniBand delivers specific advantages in the following areas:

optimization of multi-core server and storage power and performance;
efficient capacity scaling in native OS and virtualized environments;
end-to-end Ethernet and Fibre Channel connectivity; and
superior price performance, leading to savings in capital expenditures and operating expenses.

Today's servers are dominated by multicore architectures. Two quad-core CPUs can consume up to 240 watts of power. Both CPU and memory access bus architectures today support 80 Gbps of bandwidth, enabling memory usage of up to 8GB, which consumes about 96 watts of power. Connecting this high-performance compute system to the outside world can be accomplished via power-efficient single or dual Gigabit Ethernet ports (4 watts). With network-intensive applications running on the CPU and memory subsystems, however, they can become bottlenecked in the I/O with Gigabit Ethernet delivering only 1 Gbps of bandwidth and 50 microseconds of application latency. This results in idle CPU and memory cycles and, therefore, wasted power and energy costs.

The problem of mismatched I/O and bandwidth is further exacerbated in virtualized server environments, where the execution of multiple virtual machines per physical server demands significantly higher I/O throughput. In these environments, four to six Gigabit Ethernet network interface cards (NICs) and two Fibre Channel host bus adapters (HBAs) are commonly used to deliver about 12 Gbps per server. Using multiple I/O adapters somewhat alleviates the I/O bandwidth bottleneck, but application latency remains at 50 microseconds.

Using multiple I/O adapters in the server, however, brings new challenges in the areas of I/O utilization, cost and power. The I/O capacity provisioned in each server may be underutilized, because the individual adapters serving specific functions cannot be repurposed for SAN, LAN and IPC functions on demand. Moreover, this scenario uses about 28 watts of power and adds additional costs for more complex cabling.

Because InfiniBand delivers all of the services needed for unified I/O, one single or dual-port InfiniBand adapter can replace multiple adapters. When InfiniBand is used instead of Gigabit Ethernet or a mixed technology scenario, it offers 10 Gbps to 40 Gbps of bandwidth supported using PCI Express 1.x and 2.x, thereby realizing the full power of the compute and memory systems, and avoiding idle cycles for network I/O-sensitive applications.

With this solution, I/O power is reduced to 10 watts, and application latency is brought down from 50 microseconds to about one microsecond. In addition, I/O utilization is significantly improved, because InfiniBand's channelized I/O (where the I/O adapter resources are partitioned and can be dedicated to applications or virtual machines) and quality-of-services features enable repurposing of available I/O adapter bandwidth for different functions. This solution also enables the use of one wire to support all I/O from the servers, significantly reducing cabling management complexity.

When InfiniBand is used to deliver a unified I/O solution, it eliminates capacity scaling complexities. The key is the separation of compute, LAN and SAN capacity management, where each of the three islands of infrastructure can be independently scaled and managed by leveraging virtualized Ethernet and Fibre Channel ports available on InfiniBand-to-Ethernet and InfiniBand-to-Fibre Channel gateway systems.

The significant benefit of the InfiniBand solution is a savings of up to 50 percent in CAPEX and OPEX. The same high-bandwidth InfiniBand adapter is used for compute, LAN and SAN traffic. I/O resources can be provisioned in each adapter to provide the same segmentation as when multiple adapters are used, but with the added flexibility that those resources can be repurposed on demand. This also eliminates the cost of underutilized NICs and HBAs, as well as cabling complexity.

Servers using InfiniBand adapters expose IP/Ethernet interfaces to applications for LAN traffic. Similarly, they expose SCSI or iSCSI interfaces to applications for SAN traffic. This applies to both native OS and virtualized server environments. InfiniBand-to-Ethernet and InfiniBand-to-Fibre Channel gateway solutions provide virtualized Ethernet and FC ports for connectivity to existing LAN and FC SAN infrastructures. Together, seamless end-to-end Ethernet and Fibre Channel connectivity is achieved, while improving performance and utilization.

Sujal Das is the senior director of product management at Mellanox Technologies, Santa Clara, Calif.

For more information (click here)