1################################################################################ 2# # 3# NFS/RDMA README # 4# # 5################################################################################ 6 7 Author: NetApp and Open Grid Computing 8 Date: May 29, 2008 9 10Table of Contents 11~~~~~~~~~~~~~~~~~ 12 - Overview 13 - Getting Help 14 - Installation 15 - Check RDMA and NFS Setup 16 - NFS/RDMA Setup 17 18Overview 19~~~~~~~~ 20 21 This document describes how to install and setup the Linux NFS/RDMA client 22 and server software. 23 24 The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server 25 was first included in the following release, Linux 2.6.25. 26 27 In our testing, we have obtained excellent performance results (full 10Gbit 28 wire bandwidth at minimal client CPU) under many workloads. The code passes 29 the full Connectathon test suite and operates over both Infiniband and iWARP 30 RDMA adapters. 31 32Getting Help 33~~~~~~~~~~~~ 34 35 If you get stuck, you can ask questions on the 36 37 nfs-rdma-devel@lists.sourceforge.net 38 39 mailing list. 40 41Installation 42~~~~~~~~~~~~ 43 44 These instructions are a step by step guide to building a machine for 45 use with NFS/RDMA. 46 47 - Install an RDMA device 48 49 Any device supported by the drivers in drivers/infiniband/hw is acceptable. 50 51 Testing has been performed using several Mellanox-based IB cards, the 52 Ammasso AMS1100 iWARP adapter, and the Chelsio cxgb3 iWARP adapter. 53 54 - Install a Linux distribution and tools 55 56 The first kernel release to contain both the NFS/RDMA client and server was 57 Linux 2.6.25 Therefore, a distribution compatible with this and subsequent 58 Linux kernel release should be installed. 59 60 The procedures described in this document have been tested with 61 distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). 62 63 - Install nfs-utils-1.1.2 or greater on the client 64 65 An NFS/RDMA mount point can be obtained by using the mount.nfs command in 66 nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils 67 version with support for NFS/RDMA mounts, but for various reasons we 68 recommend using nfs-utils-1.1.2 or greater). To see which version of 69 mount.nfs you are using, type: 70 71 $ /sbin/mount.nfs -V 72 73 If the version is less than 1.1.2 or the command does not exist, 74 you should install the latest version of nfs-utils. 75 76 Download the latest package from: 77 78 http://www.kernel.org/pub/linux/utils/nfs 79 80 Uncompress the package and follow the installation instructions. 81 82 If you will not need the idmapper and gssd executables (you do not need 83 these to create an NFS/RDMA enabled mount command), the installation 84 process can be simplified by disabling these features when running 85 configure: 86 87 $ ./configure --disable-gss --disable-nfsv4 88 89 To build nfs-utils you will need the tcp_wrappers package installed. For 90 more information on this see the package's README and INSTALL files. 91 92 After building the nfs-utils package, there will be a mount.nfs binary in 93 the utils/mount directory. This binary can be used to initiate NFS v2, v3, 94 or v4 mounts. To initiate a v4 mount, the binary must be called 95 mount.nfs4. The standard technique is to create a symlink called 96 mount.nfs4 to mount.nfs. 97 98 This mount.nfs binary should be installed at /sbin/mount.nfs as follows: 99 100 $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs 101 102 In this location, mount.nfs will be invoked automatically for NFS mounts 103 by the system mount command. 104 105 NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed 106 on the NFS client machine. You do not need this specific version of 107 nfs-utils on the server. Furthermore, only the mount.nfs command from 108 nfs-utils-1.1.2 is needed on the client. 109 110 - Install a Linux kernel with NFS/RDMA 111 112 The NFS/RDMA client and server are both included in the mainline Linux 113 kernel version 2.6.25 and later. This and other versions of the 2.6 Linux 114 kernel can be found at: 115 116 ftp://ftp.kernel.org/pub/linux/kernel/v2.6/ 117 118 Download the sources and place them in an appropriate location. 119 120 - Configure the RDMA stack 121 122 Make sure your kernel configuration has RDMA support enabled. Under 123 Device Drivers -> InfiniBand support, update the kernel configuration 124 to enable InfiniBand support [NOTE: the option name is misleading. Enabling 125 InfiniBand support is required for all RDMA devices (IB, iWARP, etc.)]. 126 127 Enable the appropriate IB HCA support (mlx4, mthca, ehca, ipath, etc.) or 128 iWARP adapter support (amso, cxgb3, etc.). 129 130 If you are using InfiniBand, be sure to enable IP-over-InfiniBand support. 131 132 - Configure the NFS client and server 133 134 Your kernel configuration must also have NFS file system support and/or 135 NFS server support enabled. These and other NFS related configuration 136 options can be found under File Systems -> Network File Systems. 137 138 - Build, install, reboot 139 140 The NFS/RDMA code will be enabled automatically if NFS and RDMA 141 are turned on. The NFS/RDMA client and server are configured via the 142 SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER config options that both 143 depend on SUNRPC and INFINIBAND. The default value of both options will be: 144 145 - N if either SUNRPC or INFINIBAND are N, in this case the NFS/RDMA client 146 and server will not be built 147 - M if both SUNRPC and INFINIBAND are on (M or Y) and at least one is M, 148 in this case the NFS/RDMA client and server will be built as modules 149 - Y if both SUNRPC and INFINIBAND are Y, in this case the NFS/RDMA client 150 and server will be built into the kernel 151 152 Therefore, if you have followed the steps above and turned no NFS and RDMA, 153 the NFS/RDMA client and server will be built. 154 155 Build a new kernel, install it, boot it. 156 157Check RDMA and NFS Setup 158~~~~~~~~~~~~~~~~~~~~~~~~ 159 160 Before configuring the NFS/RDMA software, it is a good idea to test 161 your new kernel to ensure that the kernel is working correctly. 162 In particular, it is a good idea to verify that the RDMA stack 163 is functioning as expected and standard NFS over TCP/IP and/or UDP/IP 164 is working properly. 165 166 - Check RDMA Setup 167 168 If you built the RDMA components as modules, load them at 169 this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel 170 card: 171 172 $ modprobe ib_mthca 173 $ modprobe ib_ipoib 174 175 If you are using InfiniBand, make sure there is a Subnet Manager (SM) 176 running on the network. If your IB switch has an embedded SM, you can 177 use it. Otherwise, you will need to run an SM, such as OpenSM, on one 178 of your end nodes. 179 180 If an SM is running on your network, you should see the following: 181 182 $ cat /sys/class/infiniband/driverX/ports/1/state 183 4: ACTIVE 184 185 where driverX is mthca0, ipath5, ehca3, etc. 186 187 To further test the InfiniBand software stack, use IPoIB (this 188 assumes you have two IB hosts named host1 and host2): 189 190 host1$ ip link set dev ib0 up 191 host1$ ip address add dev ib0 a.b.c.x 192 host2$ ip link set dev ib0 up 193 host2$ ip address add dev ib0 a.b.c.y 194 host1$ ping a.b.c.y 195 host2$ ping a.b.c.x 196 197 For other device types, follow the appropriate procedures. 198 199 - Check NFS Setup 200 201 For the NFS components enabled above (client and/or server), 202 test their functionality over standard Ethernet using TCP/IP or UDP/IP. 203 204NFS/RDMA Setup 205~~~~~~~~~~~~~~ 206 207 We recommend that you use two machines, one to act as the client and 208 one to act as the server. 209 210 One time configuration: 211 212 - On the server system, configure the /etc/exports file and 213 start the NFS/RDMA server. 214 215 Exports entries with the following formats have been tested: 216 217 /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash) 218 /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) 219 220 The IP address(es) is(are) the client's IPoIB address for an InfiniBand 221 HCA or the cleint's iWARP address(es) for an RNIC. 222 223 NOTE: The "insecure" option must be used because the NFS/RDMA client does 224 not use a reserved port. 225 226 Each time a machine boots: 227 228 - Load and configure the RDMA drivers 229 230 For InfiniBand using a Mellanox adapter: 231 232 $ modprobe ib_mthca 233 $ modprobe ib_ipoib 234 $ ip li set dev ib0 up 235 $ ip addr add dev ib0 a.b.c.d 236 237 NOTE: use unique addresses for the client and server 238 239 - Start the NFS server 240 241 If the NFS/RDMA server was built as a module 242 (CONFIG_SUNRPC_XPRT_RDMA_SERVER=m in kernel config), load the RDMA 243 transport module: 244 245 $ modprobe svcrdma 246 247 Regardless of how the server was built (module or built-in), start the 248 server: 249 250 $ /etc/init.d/nfs start 251 252 or 253 254 $ service nfs start 255 256 Instruct the server to listen on the RDMA transport: 257 258 $ echo rdma 20049 > /proc/fs/nfsd/portlist 259 260 - On the client system 261 262 If the NFS/RDMA client was built as a module 263 (CONFIG_SUNRPC_XPRT_RDMA_CLIENT=m in kernel config), load the RDMA client 264 module: 265 266 $ modprobe xprtrdma.ko 267 268 Regardless of how the client was built (module or built-in), use this 269 command to mount the NFS/RDMA server: 270 271 $ mount -o rdma,port=20049 <IPoIB-server-name-or-address>:/<export> /mnt 272 273 To verify that the mount is using RDMA, run "cat /proc/mounts" and check 274 the "proto" field for the given mount. 275 276 Congratulations! You're using NFS/RDMA! 277