Making Lustre work with Xen on RHEL

Problem scenario: I want to make Xen Dom0 work as a Lustre client. I am using RHEL 5.3, Xen 3.1 and running a "vmlinuz-2.6.18-128.el5xen" kernel on a x86_64 platform.

I looked up a bunch of documents, and it turned out the only one that worked for me was https://wiki.heprc.uvic.ca/twiki/bin/view/Main/LustrePlusXen. This post is mostly the same as that link, only with more details.

Followed the hint from http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg06167.html, I tried to only build the Lustre client source directly on Xen but didn't succeed. A more complicated document can be found at http://trac.nchc.org.tw/grid/wiki/Xen_Lustre, but I got stuck when trying to build a new version of Xen image.

And here are the steps that worked for me:

0. Make sure the e2fsprogs package is installed and up to date. It can be downloaded from the Lustre download pages at http://www.sun.com/software/products/lustre/get.jsp. Choose the right platform, get the rpm, and install it with root; or update it if you already have it on your machine:

$ rpm -ivh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm
or
$ rpm -Uvh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm

1. Download the right version of Luster patched kenel source rpm and the Lustre source rpm from http://www.oracle.com/technology/software/sun_az_index.html#L, according to your kernel version and platform. To me it is kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm and lustre-source-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm. Copy them to your home directory.

2. Create the "rpmbuild" directories in your home directory, and install the Lustre patched kernel source rpm:

$ cd ~
$ mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
$ echo '%_topdir /home/{username}/rpmbuild' >./.rpmmacros
$ rpm -i kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm

3. Build the source RPMS into the complete set of kernel RPMS:

$ rpmbuild -bb --without kabichk --target=x86_64 rpmbuild/SPECS/kernel-2.6.spec

If a dependency problem about unifdef is encountered during the building process, try to install it with "yum install unifdef" and then build again.

4. The rpms built will be located in rpmbuild/RPMS/x86_64/, go there and install the kernel-lustre-xen and kernel-lustre-xen-devel rpm as root:

$ cd rpmbuild/RPMS/x86_64/
$ su
$ rpm -ivh kernel-lustre-xen-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
$ rpm -ivh kernel-lustre-xen-devel-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

5. Go back to your home directory and install the Lustre source rpm as root. This will go to /usr/src by default:

$ cd /home/{username}
$ rpm -ivh lustre-source-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

Again there may be a dependency problem about expect reported. If that happens, try to install it with "yum install expect" and then install again.

6. Go to /usr/src/lustre-1.8.1.1 as root, configure it by pointing to the right kernel source directory, and make the rpms:

$ cd /usr/src/lustre-1.8.1.1
$ ./configure --with-linux=/usr/src/kernels/2.6.18-128.7.1.el5_lustre.1.8.1.1-xen-x86_64
$ make rpms

7. At the end of the "make" output it prints where the rpms are written to. Go there and install the lustre-modules, lustre-ldiskfs, and lustre rpms:

$ cd /usr/src/redhat/RPMS/x86_64/
$ rpm -ivh lustre-modules-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210149.x86_64.rpm
$ rpm -ivh lustre-ldiskfs-3.0.9-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210150.x86_64.rpm
$ rpm -ivh lustre-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210149.x86_64.rpm

8. Open your grub.conf, you should see a section for the "xen.gz-2.6.18-128.7.1.el5_lustre.1.8.1.1" kernel; modify the default boot kernel image to this one and reboot:

$ vim /etc/grub.conf
$ reboot

9. After the reboot, you should be able to use the Dom0 as a Lustre MDS, MGS, OSS, or client. I'm not very sure about how to make a DomU work with Lustre, but I guess the only necessary extra steps will be (1) configure the DomU instance to use the "xen*lustre*" kernel, and (2) install the same set of rpms as in step 7 to the DomU instance once it is up running.