PCI Express features power I/O virtualization

PCI Express features power I/O virtualization

The PCI-SIG has defined virtualization technology based on PCIe (SR-IOV and MR-IOV), and industry leaders are developing PCIe switches that take full advantage of emerging I/O virtualization technology

BY MIGUEL RODRIGUEZ
PLX Technology, Sunnyvale, CA
http://www.plxtech.com

I/O virtualization is a concept that has interested designers for several years. For some time now, server vendors have been looking for ways to maximize system resources efficiently by detaching the physical function of an I/O device from the one server and virtualizing its resources to make them available to multiple servers. The virtualized endpoint appears as a dedicated resource to each of the servers. Although the benefits of a virtualized system are well understood, its implementation has proven to be difficult.

Consequently, a number of vendors provide limited and specialized solutions in an effort to solve the virtualization needs. Most of these solutions, however, have traditionally been application specific, with several variations between vendors.

Furthermore, these solutions have been in large software-burdened implementations, with high overhead resulting in lower-performing systems. As a result, industry efforts are taking place now focused on providing a standardized solution for the I/O virtualization problem. The major technologies considered are Ethernet, InfiniBand, and PCI Express (PCIe), each of which has its own set of virtues and flaws.

Although the time frame for the industry to determine a solution is still unknown, this article will focus on some the efforts which have taken place in PCIe to address this need. Specifically, we’ll look at features added to the PCI Express base specification to provide hardware hooks (assistance) so that users can provide their own levels of support. These features include Access Control Services (ACS) and Alternative Routing-ID Interpretation (ARI) which are used to enhance the capabilities of PCIe endpoints, as well as take advantage of other advanced features found in PCIe endpoints, such as Single Root-I/O Virtualization.

The need for I/O virtualization

A typical server consists of a CPU with a number of dedicated I/O devices. Anywhere from a couple to 14 of these servers can be connected in a cluster, each with its own dedicated I/O resources.

These resources provide the local CPU with data from the outside world (for processing, etc). Typical I/O devices include Ethernet controllers (NICs) for communicating with other servers and storage network controllers (such as Fibre Channel adapters) for shared storage pool between servers.

When in use, the server takes advantage of the capabilities of the NIC and/or HBA devices running at full bandwidth while accessing data. However, these accesses are not sustained all the time, but instead are performed in a “bursty” manner, which means that the I/O devices on the servers spend a large amount of time idle, resulting in device and power inefficiencies. Virtualization of the endpoint allows its resources to be used by multiple entities, increasing the efficiency of the device while reducing the overall cost and power.

Fig. 1. Blade chassis with dedicated I/O.

Virtualization solutions with PCIe

The PCI-SIG is working on solutions to address the industry’s virtualization needs. It has released two specifications targeting two virtualization usage models with both solutions based on PCIe.

The first is called Single Root IO Virtualization or SR-IOV. In the SR-IOV model, the resources of a given PCIe endpoint can be shared within a number of guest operating systems running on virtual machines (VM). A VM presents the guest OS with an abstraction of a virtual processor, allowing it to execute directly on a logical processor while retaining selective control of the processor and its resources (for example, memory and interrupts), as well as I/O.

As the name Single Root implies, in this model there is a single CPU entity which can consist of multiple CPUs, multiple cores, or both. Each guest OS operates independently of one another, and they all share the same interface to all physical devices, such as processor, memory, I/O, and other peripherals. The software stack (hypervisor) ensures that each guest OS behaves as a nonvirtualized platform.

The second usage model for virtualization available from the PCI-SIG is called Multi-Root IO Virtualization (MR-IOV). MR-IOV expands on the capabilities of SR-IOV. In MR-IOV there can be multiple CPU entities in one system instead of only one, as is in the case of SR-IOV. Mechanisms for sharing the I/O resources between multiple CPU entities are defined in the MR-IOV specification available from the PCI-SIG.

PCIe switches support ACS, ARI

SR-IOV defines mechanisms for a system’s endpoints and its CPU to allow sharing of its resources. This model was designed to be transparent to existing PCIe switches available in the market today.

Industry leaders, such as PLX Technology, have implemented optional features defined in the PCI Express Base Specification to further assist the deployment of SR-IOV–enabled systems. These features include Access Control Services (ACS) and Alternate Routing-ID Interpretation (ARI). ACS in a PCIe switch allows the system to have additional control over peer-to-peer transactions.

A peer-to-peer transaction in a PCIe switch can be classified as any transaction initiated from a given port to any other port without targeting the upstream port of a switch, in other words, without involving the CPU to route the transaction. In an ACS-enabled system, such transactions are not allowed, but instead all transactions are forced in the upstream direction to the CPU to be further qualified before being rerouted down to their destinations. This capability provides additional security mechanisms, which can prevent unwanted communication between two or more different functions within a single I/O device.

ARI extends the capabilities of a PCIe endpoint by increasing the number of available device functions from eight up to 256 by using the Bus and Device bit fields from the requester ID. A system needing to support ARI requires all devices in the PCIe chain (CPU, PCIe Switch, endpoint) to support ARI.

Consequently, the PCIe switch between the CPU and the endpoint needs to be able to decode and route packets accordingly. Without ARI, a virtual system cannot take advantage of the additional functions enabled in the PCIe endpoint. In a virtualized system, 16 function are typically available with some endpoints implementing as many as 256.

Fig. 2. Blade chassis with multiple VMs per blade.

Conclusion

System designers’ interest in virtualization continues to grow. The industry in general sees the need for virtual systems and understands the advantages: lower overall cost of a data center through reducing the number of I/O devices, resulting in higher efficiency and lower power and cost.

On this front, the PCI-SIG has defined virtualization technology based on PCIe (SR-IOV and MR-IOV). Industry leaders such as PLX Technology, whose Gen2 product portfolio supports ACS and ARI, are developing PCIe switches that take full advantage of emerging I/O virtualization technology. ■