IWARP
Encyclopedia
The Internet Wide Area RDMA Protocol (iWARP) is a computer network
ing protocol for transferring data efficiently.
It is sometimes referred to simply as "RDMA", though RDMA is not a feature exclusive to iWARP.
(IETF) published the RDMA Consortium's RDMA over Transmission Control Protocol
(TCP) standard. iWARP is a superset of the Virtual Interface Architecture
(VIA) that permits zero-copy
transmission over TCP and SCTP. It is often compared to InfiniBand
, also based on VIA, and distinguishes itself by being an implementation on top of IP networks (typically Ethernet), rather than specialized hardware.
However, TCP does not respect message boundaries; it sends data as a sequence of bytes without regard to protocol data units (PDU). In this regard, DDP itself may be better suited for SCTP, and indeed the IETF proposed a standard RDMA over SCTP. To run DDP over TCP requires a tweak known as marker PDU aligned (MPA) framing to guarantee boundaries of messages.
Furthermore, DDP is not intended to be accessed directly. Instead, a separate RDMA protocol (RDMAP) provides the services to read and write data. Therefore, the entire RDMA over TCP specification is really RDMAP over DDP over MPA over TCP. All of these protocols can be implemented in hardware.
Unlike IB, iWARP only has reliable connected communication as this is the only service that TCP and SCTP provide. The iWARP specification also omits many of the special features of IB, such as atomic remote operations.
(rNICs). As simple data losses are rare in tightly coupled network environments, the error-correction mechanisms of TCP may be performed by software while the more frequently performed communications are handled strictly by logic embedded on the rNIC. Similarly, connections are often established entirely by software and then handed off to the hardware. Furthermore, the handling of iWARP specific protocol details is often isolated from the TCP implementation, allowing rNICs to be used for both as RDMA offload and TCP offload (in support of traditional sockets based TCP/IP applications). The portion of the hardware implementation used for implementing the TCP protocol is known as the TCP Offload Engine (TOE).
TOE itself does not prevent copying on the receive side, and must be combined with RDMA hardware for zero-copy results. The RDMA / TCP specification is a set of different wire protocols intended to be implemented in hardware (though it seems feasible to emulate it in software for compatibility but without the performance benefits).
Several programmatic interfaces have been proposed, and implemented in different forms, including uDAPL, kDAPL, IT-API, RNICPI, and Open Fabrics Verbs. Similarly, several implementations of these interfaces are available for different platforms, including Windows and Linux.
for Linux
operating systems, and the Winsock Direct protocol for Microsoft Windows
.
Computer network
A computer network, often simply referred to as a network, is a collection of hardware components and computers interconnected by communication channels that allow sharing of resources and information....
ing protocol for transferring data efficiently.
It is sometimes referred to simply as "RDMA", though RDMA is not a feature exclusive to iWARP.
History
In 2007, the Internet Engineering Task ForceInternet Engineering Task Force
The Internet Engineering Task Force develops and promotes Internet standards, cooperating closely with the W3C and ISO/IEC standards bodies and dealing in particular with standards of the TCP/IP and Internet protocol suite...
(IETF) published the RDMA Consortium's RDMA over Transmission Control Protocol
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
(TCP) standard. iWARP is a superset of the Virtual Interface Architecture
Virtual Interface Architecture
The Virtual Interface Architecture is an abstract model of a user-level zero-copy network, and is the basis for InfiniBand, iWARP and RoCE...
(VIA) that permits zero-copy
Zero-copy
"Zero-copy" describes computer operations in which the CPU does not perform the task of copying data from one memory area to another. This is most often used to save on processing power and memory use when sending files over a network.- Principle :...
transmission over TCP and SCTP. It is often compared to InfiniBand
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
, also based on VIA, and distinguishes itself by being an implementation on top of IP networks (typically Ethernet), rather than specialized hardware.
Protocol
The main component the iWARP protocol is the Direct Data Placement Protocol (DDP), which permits the actual zero-copy transmission. DDP itself does not perform the transmission; the underlying protocol (TCP or SCTP) does.However, TCP does not respect message boundaries; it sends data as a sequence of bytes without regard to protocol data units (PDU). In this regard, DDP itself may be better suited for SCTP, and indeed the IETF proposed a standard RDMA over SCTP. To run DDP over TCP requires a tweak known as marker PDU aligned (MPA) framing to guarantee boundaries of messages.
Furthermore, DDP is not intended to be accessed directly. Instead, a separate RDMA protocol (RDMAP) provides the services to read and write data. Therefore, the entire RDMA over TCP specification is really RDMAP over DDP over MPA over TCP. All of these protocols can be implemented in hardware.
Unlike IB, iWARP only has reliable connected communication as this is the only service that TCP and SCTP provide. The iWARP specification also omits many of the special features of IB, such as atomic remote operations.
Implementation
Because a kernel implementation of the TCP stack can be seen as a bottleneck, the protocol is typically implemented in hardware RDMA network interface controllersNetwork card
A network interface controller is a computer hardware component that connects a computer to a computer network....
(rNICs). As simple data losses are rare in tightly coupled network environments, the error-correction mechanisms of TCP may be performed by software while the more frequently performed communications are handled strictly by logic embedded on the rNIC. Similarly, connections are often established entirely by software and then handed off to the hardware. Furthermore, the handling of iWARP specific protocol details is often isolated from the TCP implementation, allowing rNICs to be used for both as RDMA offload and TCP offload (in support of traditional sockets based TCP/IP applications). The portion of the hardware implementation used for implementing the TCP protocol is known as the TCP Offload Engine (TOE).
TOE itself does not prevent copying on the receive side, and must be combined with RDMA hardware for zero-copy results. The RDMA / TCP specification is a set of different wire protocols intended to be implemented in hardware (though it seems feasible to emulate it in software for compatibility but without the performance benefits).
Interfaces
iWARP is a protocol, not an implementation, but defines protocol behavior in terms of the operations that are legal for the protocol, known as Verbs. As such, iWARP does not have any single standard programming interface. However, programming interfaces tend to very closely correspond to the Verbs.Several programmatic interfaces have been proposed, and implemented in different forms, including uDAPL, kDAPL, IT-API, RNICPI, and Open Fabrics Verbs. Similarly, several implementations of these interfaces are available for different platforms, including Windows and Linux.
Services available
Networking services implemented over iWARP include those offered in the OpenFabrics Enterprise Distribution (OFED) by the OpenFabrics AllianceOpenFabrics Alliance
The OpenFabrics Alliance is a non-profit organization that evangelizes remote direct memory access switched fabric technologies for server and storage connectivity...
for Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
operating systems, and the Winsock Direct protocol for Microsoft Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
.
- NFS over RDMA
- Sockets Direct ProtocolSockets Direct ProtocolThe Sockets Direct Protocol is a networking protocol originally defined by the Software Working Group of the InfiniBand Trade Association. Originally designed for InfiniBand , SDP now has been redefined as a transport agnostic protocol for Remote Direct Memory Access network fabrics...
(SDP) - iSCSI Extensions for RDMAISCSI Extensions for RDMAThe iSCSI Extensions for RDMA is a computer network protocol that extends the Internet Small Computer System Interface protocol to use Remote Direct Memory Access...
(iSER) - SCSI RDMA ProtocolSCSI RDMA ProtocolIn computing the SCSI RDMA Protocol is a protocol that allows one computer to access SCSI devices attached to another computer via remote direct memory access . The SRP protocol is also known as the SCSI Remote Protocol. The use of RDMA makes higher throughput and lower latency possible than what...
(SRP)
External links
- iWARP Consortium at the University of New Hampshire InterOperability LaboratoryUniversity of New Hampshire InterOperability LaboratoryThe University of New Hampshire InterOperability Laboratory provides independent, broad-based interoperability and standards conformance testing for data, telecommunications and storage networking products and technologies...
-- Testing on iWARP devices - iWARP 2.0 - First Stab at a Potential Features List, a presentation given by Felix Marti on March 17th, 2010 during a meeting of the OpenFabrics AllianceOpenFabrics AllianceThe OpenFabrics Alliance is a non-profit organization that evangelizes remote direct memory access switched fabric technologies for server and storage connectivity...
. - EE Times article (dated 12/07/2001) on iWARP
- Network World article (dated 09/26/05) on iWARP and competing technologies
- Remote Direct Data Placement Charter links to IETF specifications on iWARP
- NetEffect -- producers of iWARP equipment
- iVivity -- producers of iWARP equipment
- Chelsio Communications -- producers of iWARP equipment
- Lamprey Networks - Producers of iWARP conformance test suite.
- iWARP over SCTP -- UBC group's slides from their CAC 2008 paper detailing a design of iWARP over SCTP