RDMA Enabled Apache (mod_rdma)
Description: As we depend upon the Internet more and more each day for all of lifes activities, todays web servers will soon be unable to handle the increased demand. In fact it is already common place to look toward distributed web servers to handle many moderate to large web sites. The main reason behind this is the web server simply can not keep up with generating and retrieving content, while at the same time sending that data over the network. The cost of network processing is just too big a burden to bear. The reason for this is the way TCP/IP is handled. Today's NICs simply put bits on and off of the wire, so to speak. All protocol processing is handled by the CPU. TCP Offload Engines, or TOE cards have been introduced in recent years, to lack luster success. These devices aim to move the network processing away from the CPU and to the NIC. This is a good idea, albeit not a full solution. The limiting factor of TOE cards is they do not address the problem of accesing memory. The CPU must still move data to and from the NIC, in order to do this it must make a copy of data to pass through the OS kernel, further confounding the problem.
Due to the shortcomings of TOE cards, a technique known as Remote Direct Memory Access has been developed, also known as RDMA. RDMA not only completely offloads all protocol processing to the NIC, but RDMA also frees the CPU of the burden of dealing with memory. The network adapter, or RDMA NIC (RNIC), handles movement of data directly from user space. There is no copy made, as data is not passed through the OS kernel. RDMA achieves what is known as zero-copy.
In the past RDMA has proved to be a great solution to networking problems, however it was limited to the data center use in technologies such as InfiniBand. However, with the technology known as iWARP, it is now possible to utilize RDMA over ordinary TCP networks. In other words RDMA can work over the Internet.
One of the limiting factors for RDMA deployment is it takes special hardware. While a NIC upgrade on a web server is perfectly reasonable, requiring all web clients to upgrade their NICs to an RNIC is not rational. Fortunately using what is known as software iWARP, web clients can make use of a downloadble plug in, or direct software support to emulate the iWARP protocols, and let the web server take advantages of its RNIC.
Our work in mod_rdma, is to create a module for the popular Apache web server that enables it to use RDMA. We outfit a web server with 10Gigabit iWARP hardware and utilize clients running software iWARP modified wget and apache bench programs. The result is much improved web server performance, enabling a single server to fully saturate a 10 Gigabit per second network. This web page serves as the distribution site for mod_rdma source code as well as any documentation and support. This will be the main source of public dissemination of information on RDMA Enabled Apache.
2. Modifications to Apache server
3. Apache Hooks
4. Transport Modules
5. Supported Transport Types
6. Included Client Applications
9. Author and Contact Information
Download a tarball containing mod_rdma, wget, and Apache bench (ab). Please send an email to Dennis just so we have a general idea how many people are downloading the code.
To use the optional RDMA sendfile capability you need to install RDMA sendfile libraries. Either kernel or user space. Download RDMA Sendfile. To enable mod_rdma to use RDMA sendfile look for the RDMA Sendfile note in the Makefile.
Note: Software iWARP may be of use on the client side for testing.
1. Wget Hardware iWARP - HW iWARP single client performance
2. Wget Software iWARP - HW iWARP Server and Software iWARP clients
3. AB Hardware iWARP - HW IWARP single client perofmance
4. AB Software IWARP - HW iWARP Server and Software iWARP clients
Dennis Dalessandro, and Pete Wyckoff, Accelerating Web Protocols Using RDMA, 6th IEEE International Symposium on Network Computing and Applications (NCA 07), Cambridge MA
Dennis Dalessandro, and Pete Wyckoff, Accelerating Web Protocols Using RDMA , Poster Session of Supercomputing 2006 (SC|06), Tampa, FL.
- Poster - Email for a copy (30 MB)
Dennis Dalessandro, Pete Wyckoff and Gary Montry, Initial Performance Evaluation of the NetEffect 10 Gigabit iWARP Adapter, Proceedings of the RAIT Workshop at the IEEE Cluster 2006 Conference, Barcelona, Spain.
Dennis Dalessandro, iWARP: Its Not Just For The LAN Anymore, The 2006 Commodity Cluster Symposium: On the Use of Commodity Clusters for Large-Scale Scientific Applications, Baltimore, MD.
Contact Information: Dennis Dalessandro.