Common System Interface
Encyclopedia
The Intel QuickPath Interconnect (QuickPath, QPI) is a point-to-point processor
interconnect
developed by Intel which replaces the Front Side Bus (FSB) in Xeon
, Itanium
, and certain desktop platforms. It was designed to compete with HyperTransport
. Intel first delivered it in November 2008 on the Intel Core i7-9xx
desktop processors and X58
chipset
. Intel developed QPI at its Massachusetts Microprocessor Design Center (MMDC) by members of what had been DEC's
Alpha Development Group, which Intel acquired from Compaq and HP. Prior to the name's announcement, Intel referred to it as Common System Interface (CSI). Earlier incarnations were known as YAP (Yet Another Protocol) and YAP+.
). In more complex instances of the architecture, separate QPI link pairs connect one or more processors and one or more IO hubs or routing hubs in a network on the motherboard, allowing all of the components to access other components via the network. As with HyperTransport, the QuickPath Architecture assumes that the processors will have integrated memory controller
s, and enables a non-uniform memory access
(NUMA) architecture. It was first released in Xeon processors in March 2009 and Itanium processors in February 2010.
Each QPI comprises two 20-lane point-to-point data links, one in each direction (full duplex), with a separate clock pair in each direction, for a total of 42 signals. Each signal is a differential pair
, so the total number of pins is 84. The 20 data lanes are divided onto four "quadrants" of 5 lanes each. The basic unit of transfer is the 80-bit "flit," which is transferred in two clock cycles (four 20 bit transfers, two per clock.) The 80-bit "flit" has 8 bits for error detection, 8 bits for "link-layer header," and 64 bits for "data". QPI bandwidths are advertised by computing the transfer of 64 bits (8 bytes) of data every two clock cycles in each direction.
Although the initial implementations use single four-quadrant links, the QPI specification permits other implementations. Each quadrant can be used independently. On high-reliability servers, a QPI link can operate in a degraded mode. If one or more of the 20+1 signals fails, the interface will operate using 10+1 or even 5+1 remaining signals, even reassigning the clock to a data signal if the clock fails.
The initial Nehalem implementation uses a full four-quadrant interface to achieve 25.6 GB/s, which provides exactly double the theoretical bandwidth of Intel's 1600 MHz FSB used in the X48 chipset.
Although some Core i7 processors use QPI, other Nehalem desktop and mobile processors (e.g. Core i3, Core i5, and other Core i7 processors) do not—at least in any externally accessible fashion. These processors cannot participate in a multiprocessor system. Instead, they directly implement the DMI
and PCI-e interfaces, obviating the need for a "northside" device or a processor bus of any type.
Intel describes the data throughput (in GB/s) by counting only the 64-bit data payload in each 80-bit "flit". However, Intel then doubles the result because the unidirectional send and receive link pair can be simultaneously active. Thus, Intel describes a 20-lane QPI link pair (send and receive) with a 3.2 GHz clock as having a data rate of 25.6 GB/s. A clock rate of 2.4 GHz yields a data rate of 19.2 GB/s. More generally, by this definition a two-link 20-lane QPI transfers eight bytes per clock cycle, four in each direction.
The rate is computed as follows:
Microprocessor
A microprocessor incorporates the functions of a computer's central processing unit on a single integrated circuit, or at most a few integrated circuits. It is a multipurpose, programmable device that accepts digital data as input, processes it according to instructions stored in its memory, and...
interconnect
Electrical connection
An electrical connection between discrete points allows the flow of electrons . A pair of connections is needed for a circuit.Between points with a low voltage difference, direct current can be controlled by a switch...
developed by Intel which replaces the Front Side Bus (FSB) in Xeon
Xeon
The Xeon is a brand of multiprocessing- or multi-socket-capable x86 microprocessors from Intel Corporation targeted at the non-consumer server, workstation and embedded system markets.-Overview:...
, Itanium
Itanium
Itanium is a family of 64-bit Intel microprocessors that implement the Intel Itanium architecture . Intel markets the processors for enterprise servers and high-performance computing systems...
, and certain desktop platforms. It was designed to compete with HyperTransport
HyperTransport
HyperTransport , formerly known as Lightning Data Transport , is a technology for interconnection of computer processors. It is a bidirectional serial/parallel high-bandwidth, low-latency point-to-point link that was introduced on April 2, 2001...
. Intel first delivered it in November 2008 on the Intel Core i7-9xx
Bloomfield (microprocessor)
Bloomfield is the code name for Intel high-end desktop processors sold as Core i7-9xx and single-processor servers sold as Xeon 35xx., in almost identical configurations, replacing the earlier Yorkfield processors. The Bloomfield core is closely related to the dual-processor Gainestown, which has...
desktop processors and X58
Intel X58
The Intel X58 is an Intel chip designed to connect Intel processors with Intel QuickPath Interconnect interface to peripheral devices. Supported processors implement the Nehalem microarchitecture and therefore have an integrated memory controller , so the X58 does not have a memory interface...
chipset
Chipset
A chipset, PC chipset, or chip set refers to a group of integrated circuits, or chips, that are designed to work together. They are usually marketed as a single product.- Computers :...
. Intel developed QPI at its Massachusetts Microprocessor Design Center (MMDC) by members of what had been DEC's
Digital Equipment Corporation
Digital Equipment Corporation was a major American company in the computer industry and a leading vendor of computer systems, software and peripherals from the 1960s to the 1990s...
Alpha Development Group, which Intel acquired from Compaq and HP. Prior to the name's announcement, Intel referred to it as Common System Interface (CSI). Earlier incarnations were known as YAP (Yet Another Protocol) and YAP+.
Implementation
The QPI is an element of a system architecture that Intel calls the QuickPath architecture that implements what Intel calls QuickPath technology. In its simplest form on a single-processor motherboard, a single QPI is used to connect the processor to the IO Hub (e.g., to connect an Intel Core i7 to an X58Intel X58
The Intel X58 is an Intel chip designed to connect Intel processors with Intel QuickPath Interconnect interface to peripheral devices. Supported processors implement the Nehalem microarchitecture and therefore have an integrated memory controller , so the X58 does not have a memory interface...
). In more complex instances of the architecture, separate QPI link pairs connect one or more processors and one or more IO hubs or routing hubs in a network on the motherboard, allowing all of the components to access other components via the network. As with HyperTransport, the QuickPath Architecture assumes that the processors will have integrated memory controller
Memory controller
The memory controller is a digital circuit which manages the flow of data going to and from the main memory. It can be a separate chip or integrated into another chip, such as on the die of a microprocessor...
s, and enables a non-uniform memory access
Non-Uniform Memory Access
Non-Uniform Memory Access is a computer memory design used in Multiprocessing, where the memory access time depends on the memory location relative to a processor...
(NUMA) architecture. It was first released in Xeon processors in March 2009 and Itanium processors in February 2010.
Each QPI comprises two 20-lane point-to-point data links, one in each direction (full duplex), with a separate clock pair in each direction, for a total of 42 signals. Each signal is a differential pair
Differential signaling
Differential signaling is a method of transmitting information electrically by means of two complementary signals sent on two separate wires. The technique can be used for both analog signaling, as in some audio systems, and digital signaling, as in RS-422, RS-485, Ethernet , PCI Express and USB...
, so the total number of pins is 84. The 20 data lanes are divided onto four "quadrants" of 5 lanes each. The basic unit of transfer is the 80-bit "flit," which is transferred in two clock cycles (four 20 bit transfers, two per clock.) The 80-bit "flit" has 8 bits for error detection, 8 bits for "link-layer header," and 64 bits for "data". QPI bandwidths are advertised by computing the transfer of 64 bits (8 bytes) of data every two clock cycles in each direction.
Although the initial implementations use single four-quadrant links, the QPI specification permits other implementations. Each quadrant can be used independently. On high-reliability servers, a QPI link can operate in a degraded mode. If one or more of the 20+1 signals fails, the interface will operate using 10+1 or even 5+1 remaining signals, even reassigning the clock to a data signal if the clock fails.
The initial Nehalem implementation uses a full four-quadrant interface to achieve 25.6 GB/s, which provides exactly double the theoretical bandwidth of Intel's 1600 MHz FSB used in the X48 chipset.
Although some Core i7 processors use QPI, other Nehalem desktop and mobile processors (e.g. Core i3, Core i5, and other Core i7 processors) do not—at least in any externally accessible fashion. These processors cannot participate in a multiprocessor system. Instead, they directly implement the DMI
Direct Media Interface
The Direct Media Interface is the link between an Intel northbridge and an Intel southbridge on a computer motherboard. It was first used between the 9xx chipsets and the ICH6, released in 2004. Previous chipsets had used the Hub Interface to perform the same function. Server chipsets use a...
and PCI-e interfaces, obviating the need for a "northside" device or a processor bus of any type.
QuickPath Interconnect frequency specifications
QPI operates at a clock rate of either 2.4 GHz, 2.93 GHz, or 3.2 GHz. The clock rate for a particular link depends on the capabilities of the components at each end of the link and the signal characteristics of the signal path on the printed circuit board. The non-extreme Core i7 9xx processors are restricted to a 2.4 GHz frequency at stock reference clocks. Bit transfers occur on both the rising and the falling edges of the clock, so the transfer rate is double the clock rate.Intel describes the data throughput (in GB/s) by counting only the 64-bit data payload in each 80-bit "flit". However, Intel then doubles the result because the unidirectional send and receive link pair can be simultaneously active. Thus, Intel describes a 20-lane QPI link pair (send and receive) with a 3.2 GHz clock as having a data rate of 25.6 GB/s. A clock rate of 2.4 GHz yields a data rate of 19.2 GB/s. More generally, by this definition a two-link 20-lane QPI transfers eight bytes per clock cycle, four in each direction.
The rate is computed as follows:
- 3.2 GHz
- × 2 bits/Hz (double data rate)
- × 20 (QPI link width)
- × (64/80) (data bits/flit bits)
- × 2 (unidirectional send and receive operating simultaneously)
- ÷ 8 (bits/byte)
- = 25.6 GB/s
Protocol layers
QPI is specified as a five-layer architecture, with separate physical, link, routing, transport, and protocol layers. In devices intended only for point-to-point QPI use with no forwarding, such as the Core i7-9xx and Xeon DP processors, the transport layer is not present and the routing layer is minimal.Physical layer
The physical layer comprises the actual wiring and the differential transmitters and receivers, plus the lowest-level logic that transmits and receives the physical-layer unit. The physical-layer unit is the 20-bit "phit." The physical layer transmits a 20-bit "phit" using a single clock edge on 20 lanes when all 20 lanes are available, or on 10 or 5 lanes when the QPI is reconfigured due to a failure. Note that in addition to the data signals, a clock signal is forwarded from the transmitter to receiver (which simplifies clock recovery at the expense of additional pins).Link layer
The link layer is responsible for sending and receiving 80-bit flits. Each flit is sent to the physical layer as four 20-bit phits. Each flit contains an 8-bit CRC generated by the link layer transmitter and a 72-bit payload. If the link layer receiver detects a CRC error, the receiver notifies the transmitter via a flit on the return link of the pair and the transmitter resends the flit. The link layer implements flow control using a credit/debit scheme to prevent the receiver's buffer from overflowing. The link layer supports six different classes of message to permit the higher layers to distinguish data flits from non-data messages primarily for maintenance of cache coherence. In complex implementations of the QuickPath architecture, the link layer can be configured to maintain separate flows and flow control for the different classes. It is not clear if this is needed or implemented for single-processor and dual-processor implementations.Routing layer
The routing layer sends a 72-bit unit consisting of an 8-bit header and a 64-bit payload. The header contains the destination and the message type. When the routing layer receives a unit, it examines its routing tables to determine if the unit has reached its destination. If so it is delivered to the next-higher layer. If not, it is sent on the correct outbound QPI. On a device with only one QPI, the routing layer is minimal. For more complex implementations, the routing layer's routing tables are more complex, and are modified dynamically to avoid failed QPI links.Transport layer
The transport layer is not needed and is not present in devices that are intended for only point-to-point connections. This includes the Core i7. The transport layer sends and receives data across the QPI network from its peers on other devices that may not be directly connected (i.e., the data may have been routed through an intervening device.) the transport layer verifies that the data is complete, and if not, it requests retransmission from its peer.Protocol layer
The protocol layer sends and receives packets on behalf of the device. A typical packet is a memory cache row. The protocol layer also participates in cache coherency maintenance by sending and receiving cache coherency messages.See also
- Elastic interface busElastic interface busElastic Interface buses, abbreviated as EI bus connections, can be generalized as bus connections which are high speed interfaces that send clock signals with data.-Description:...
- Fibre ChannelFibre ChannelFibre Channel, or FC, is a gigabit-speed network technology primarily used for storage networking. Fibre Channel is standardized in the T11 Technical Committee of the InterNational Committee for Information Technology Standards , an American National Standards Institute –accredited standards...
- Front side busFront side busA front-side bus is a computer communication interface often used in computers during the 1990s and 2000s.It typically carries data between the central processing unit and a memory controller hub, known as the northbridge....
- HyperTransportHyperTransportHyperTransport , formerly known as Lightning Data Transport , is a technology for interconnection of computer processors. It is a bidirectional serial/parallel high-bandwidth, low-latency point-to-point link that was introduced on April 2, 2001...
- List of device bandwidths
- PCI ExpressPCI ExpressPCI Express , officially abbreviated as PCIe, is a computer expansion card standard designed to replace the older PCI, PCI-X, and AGP bus standards...
- RapidIORapidIOThe RapidIO architecture is a high-performance packet-switched, interconnect technology for interconnecting chips on a circuit board, and also circuit boards to each other using a backplane...