Making Lustre work with Xen on RHEL

Problem scenario: I want to make Xen Dom0 work as a Lustre client. I am using RHEL 5.3, Xen 3.1 and running a "vmlinuz-2.6.18-128.el5xen" kernel on a x86_64 platform.

I looked up a bunch of documents, and it turned out the only one that worked for me was https://wiki.heprc.uvic.ca/twiki/bin/view/Main/LustrePlusXen. This post is mostly the same as that link, only with more details.

Followed the hint from http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg06167.html, I tried to only build the Lustre client source directly on Xen but didn't succeed. A more complicated document can be found at http://trac.nchc.org.tw/grid/wiki/Xen_Lustre, but I got stuck when trying to build a new version of Xen image.

And here are the steps that worked for me:

0. Make sure the e2fsprogs package is installed and up to date. It can be downloaded from the Lustre download pages at http://www.sun.com/software/products/lustre/get.jsp. Choose the right platform, get the rpm, and install it with root; or update it if you already have it on your machine:

$ rpm -ivh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm
or
$ rpm -Uvh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm

1. Download the right version of Luster patched kenel source rpm and the Lustre source rpm from http://www.oracle.com/technology/software/sun_az_index.html#L, according to your kernel version and platform. To me it is kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm and lustre-source-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm. Copy them to your home directory.

2. Create the "rpmbuild" directories in your home directory, and install the Lustre patched kernel source rpm:

$ cd ~
$ mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
$ echo '%_topdir /home/{username}/rpmbuild' >./.rpmmacros
$ rpm -i kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm

3. Build the source RPMS into the complete set of kernel RPMS:

$ rpmbuild -bb --without kabichk --target=x86_64 rpmbuild/SPECS/kernel-2.6.spec

If a dependency problem about unifdef is encountered during the building process, try to install it with "yum install unifdef" and then build again.

4. The rpms built will be located in rpmbuild/RPMS/x86_64/, go there and install the kernel-lustre-xen and kernel-lustre-xen-devel rpm as root:

$ cd rpmbuild/RPMS/x86_64/
$ su
$ rpm -ivh kernel-lustre-xen-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
$ rpm -ivh kernel-lustre-xen-devel-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

5. Go back to your home directory and install the Lustre source rpm as root. This will go to /usr/src by default:

$ cd /home/{username}
$ rpm -ivh lustre-source-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

Again there may be a dependency problem about expect reported. If that happens, try to install it with "yum install expect" and then install again.

6. Go to /usr/src/lustre-1.8.1.1 as root, configure it by pointing to the right kernel source directory, and make the rpms:

$ cd /usr/src/lustre-1.8.1.1
$ ./configure --with-linux=/usr/src/kernels/2.6.18-128.7.1.el5_lustre.1.8.1.1-xen-x86_64
$ make rpms

7. At the end of the "make" output it prints where the rpms are written to. Go there and install the lustre-modules, lustre-ldiskfs, and lustre rpms:

$ cd /usr/src/redhat/RPMS/x86_64/
$ rpm -ivh lustre-modules-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210149.x86_64.rpm
$ rpm -ivh lustre-ldiskfs-3.0.9-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210150.x86_64.rpm
$ rpm -ivh lustre-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210149.x86_64.rpm

8. Open your grub.conf, you should see a section for the "xen.gz-2.6.18-128.7.1.el5_lustre.1.8.1.1" kernel; modify the default boot kernel image to this one and reboot:

$ vim /etc/grub.conf
$ reboot

9. After the reboot, you should be able to use the Dom0 as a Lustre MDS, MGS, OSS, or client. I'm not very sure about how to make a DomU work with Lustre, but I guess the only necessary extra steps will be (1) configure the DomU instance to use the "xen*lustre*" kernel, and (2) install the same set of rpms as in step 7 to the DomU instance once it is up running.

Disable Google map recenter function in daily RDAHMM

In response to portal users' requests, we have disabled the "automatic recenter function" of Google map in the daily RDAHMM portlet. The trick turns out to be a "internally used" feature of the GMarker.openInfoWindowHtml() function, which is not included in the official online documents:

marker.openInfoWindowHtml(htmlStr, {suppressMapPan:true});

New png plotting set-ups in daily RDAHMM service

In daily RDAHMM service, we modified some of the gnuplot scripts so that the png pictures are plotted in the following manner:

(1) All points are first connected with light gray lines;
(2) On top of the gray lines, continuous sections with the same state are plotted with lines of colors corresponding to the state;
(3) On top of the colored lines, all points are plotted with "+"-shaped points of colors corresponding to the states of each point.

De-trend and De-noise options added to daily RDAHMM service

We added de-trend and de-noise functions to the daily RDAHMM service, and these features can be turned on and off by modifying the property files of the service. Since de-noise is done on the output data of de-trend, these options can only be turned on or off together -- that is, the input data is only de-trended and de-noised in a sequence when both options are on; otherwise neither is done. We'll separate them in future work.

KML service in daily RDAHMM portlet extended for date ranges

The KML generating function in daily RDAHMM portlet is extended to support date ranges, instead of single dates. The resulted KML will have a "Folder" element which contains a series of "Document" elements corresponding to each date within that date range. Each "Document" is marked with a "TimeSpan" element, so that the whole KML can be played as an animation along the time axis.

Corresponding modifications are applied to the daily RDAHMM result service.

Due to problems with current implementation, the KML function can only support a not-too-long date range, typically shorter than two years. We will try different implementation methods to solve this problem.