New state change number vs. time plot component added to daily RDAHMM portlet

We Changed the daily RDAHMM portlet to use the "DyGraph" Javascript component to draw the state change number vs. time plot. In this plot users can select data points by moving the mouse around the plot, and can zoom in to a specific time section by selecting an area, and zoom out by double click.

Support for multiple date format added to daily RDAHMM portlet

We changed the daily RDAHMM portlet so that users can use various format to specify dates in the input boxes, including 'yyyy-mm-dd', 'yyyy/MM/dd', 'MM/dd/yyyy', 'MMMM dd, yyyy', or 'MMM dd, yyyy'.

How to create a file of a given size in Linux, fastly

dd if=/dev/zero of=[path to your file] bs=256K seek=[required size]/256K count=0

This will make a file of the given size, but does not fill the file with zero or anything, so the content of the file is undefined.

We will use this for volume creation in VBS-Lustre.

New plotting component added to daily RDAHMM

We have added a new swf plotting component to the daily RDAHMM portlet, as shown in the following picture. Compared to the old plots, this new plotting component has the following advantages:

1. better point and line connections;
2. zoom-in and zoom-out in time scale;
3. slide-though along time axis;

Making Lustre work with Xen on RHEL

Problem scenario: I want to make Xen Dom0 work as a Lustre client. I am using RHEL 5.3, Xen 3.1 and running a "vmlinuz-2.6.18-128.el5xen" kernel on a x86_64 platform.

I looked up a bunch of documents, and it turned out the only one that worked for me was https://wiki.heprc.uvic.ca/twiki/bin/view/Main/LustrePlusXen. This post is mostly the same as that link, only with more details.

Followed the hint from http://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg06167.html, I tried to only build the Lustre client source directly on Xen but didn't succeed. A more complicated document can be found at http://trac.nchc.org.tw/grid/wiki/Xen_Lustre, but I got stuck when trying to build a new version of Xen image.

And here are the steps that worked for me:

0. Make sure the e2fsprogs package is installed and up to date. It can be downloaded from the Lustre download pages at http://www.sun.com/software/products/lustre/get.jsp. Choose the right platform, get the rpm, and install it with root; or update it if you already have it on your machine:

$ rpm -ivh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm
or
$ rpm -Uvh e2fsprogs-1.41.10.sun2-0redhat.rhel5.x86_64.rpm

1. Download the right version of Luster patched kenel source rpm and the Lustre source rpm from http://www.oracle.com/technology/software/sun_az_index.html#L, according to your kernel version and platform. To me it is kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm and lustre-source-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm. Copy them to your home directory.

2. Create the "rpmbuild" directories in your home directory, and install the Lustre patched kernel source rpm:

$ cd ~
$ mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
$ echo '%_topdir /home/{username}/rpmbuild' >./.rpmmacros
$ rpm -i kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm

3. Build the source RPMS into the complete set of kernel RPMS:

$ rpmbuild -bb --without kabichk --target=x86_64 rpmbuild/SPECS/kernel-2.6.spec

If a dependency problem about unifdef is encountered during the building process, try to install it with "yum install unifdef" and then build again.

4. The rpms built will be located in rpmbuild/RPMS/x86_64/, go there and install the kernel-lustre-xen and kernel-lustre-xen-devel rpm as root:

$ cd rpmbuild/RPMS/x86_64/
$ su
$ rpm -ivh kernel-lustre-xen-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
$ rpm -ivh kernel-lustre-xen-devel-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

5. Go back to your home directory and install the Lustre source rpm as root. This will go to /usr/src by default:

$ cd /home/{username}
$ rpm -ivh lustre-source-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

Again there may be a dependency problem about expect reported. If that happens, try to install it with "yum install expect" and then install again.

6. Go to /usr/src/lustre-1.8.1.1 as root, configure it by pointing to the right kernel source directory, and make the rpms:

$ cd /usr/src/lustre-1.8.1.1
$ ./configure --with-linux=/usr/src/kernels/2.6.18-128.7.1.el5_lustre.1.8.1.1-xen-x86_64
$ make rpms

7. At the end of the "make" output it prints where the rpms are written to. Go there and install the lustre-modules, lustre-ldiskfs, and lustre rpms:

$ cd /usr/src/redhat/RPMS/x86_64/
$ rpm -ivh lustre-modules-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210149.x86_64.rpm
$ rpm -ivh lustre-ldiskfs-3.0.9-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210150.x86_64.rpm
$ rpm -ivh lustre-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1xen_201007210149.x86_64.rpm

8. Open your grub.conf, you should see a section for the "xen.gz-2.6.18-128.7.1.el5_lustre.1.8.1.1" kernel; modify the default boot kernel image to this one and reboot:

$ vim /etc/grub.conf
$ reboot

9. After the reboot, you should be able to use the Dom0 as a Lustre MDS, MGS, OSS, or client. I'm not very sure about how to make a DomU work with Lustre, but I guess the only necessary extra steps will be (1) configure the DomU instance to use the "xen*lustre*" kernel, and (2) install the same set of rpms as in step 7 to the DomU instance once it is up running.

Disable Google map recenter function in daily RDAHMM

In response to portal users' requests, we have disabled the "automatic recenter function" of Google map in the daily RDAHMM portlet. The trick turns out to be a "internally used" feature of the GMarker.openInfoWindowHtml() function, which is not included in the official online documents:

marker.openInfoWindowHtml(htmlStr, {suppressMapPan:true});

New png plotting set-ups in daily RDAHMM service

In daily RDAHMM service, we modified some of the gnuplot scripts so that the png pictures are plotted in the following manner:

(1) All points are first connected with light gray lines;
(2) On top of the gray lines, continuous sections with the same state are plotted with lines of colors corresponding to the state;
(3) On top of the colored lines, all points are plotted with "+"-shaped points of colors corresponding to the states of each point.

De-trend and De-noise options added to daily RDAHMM service

We added de-trend and de-noise functions to the daily RDAHMM service, and these features can be turned on and off by modifying the property files of the service. Since de-noise is done on the output data of de-trend, these options can only be turned on or off together -- that is, the input data is only de-trended and de-noised in a sequence when both options are on; otherwise neither is done. We'll separate them in future work.

KML service in daily RDAHMM portlet extended for date ranges

The KML generating function in daily RDAHMM portlet is extended to support date ranges, instead of single dates. The resulted KML will have a "Folder" element which contains a series of "Document" elements corresponding to each date within that date range. Each "Document" is marked with a "TimeSpan" element, so that the whole KML can be played as an animation along the time axis.

Corresponding modifications are applied to the daily RDAHMM result service.

Due to problems with current implementation, the KML function can only support a not-too-long date range, typically shorter than two years. We will try different implementation methods to solve this problem.

iSCSI bug when initiator and target are on the same machine?

In order to test the pure iSCSI processing overhead, I tried to export a 5GB iSCSI target and login to it from the same machine, so that I can compare the performance of this locally connected iSCSI virtual device with that of the physical device.

I formatted the virtual device with the ext3 file system, mounted it to a directory, and ran bonnie++ under that directory to test its performance. I used file sizes of 1GB, 2GB, and 4GB (1GB is twice as large as the memory, which is the minimum required by bonnie++), and came across various errors during the test, some are the write(2) system call errors and some even reported the file system became "read only" after writing some files. The kernel also reported a journal commit I/O error on the virtual device.

The tests can be completed when the physical device is directly mounted. So I'm guessing this is a bug of iSCSI with locally connected sessions?

KML generation function added to daily RDAHMM result service

We added a new function to the daily RDAHMM result service, which can generate a KML file describing the states of all stations for a specified date. The file can be opened and shown successfully with Google Earth, but Google map has problems when showing the interactive plots.

A "Get KML For This Day" button is added to the daily RDAHMM portlet, which when clicked will call the daily RDAHMM result service to create a KML for the selected date, and show the content in a new window.

Major changes to daily RDAHMM service

We spent a lot of time re-writing a major part of the daily RDAHMM service, and here are some important changes:

1. A large amount of dirty codes are cleared up. The DailyRDAHMMService class is replaced with a new DailyRDAHMMStation class, which performs all functions DailyRDAHMMService used to complete, but with better flexibility and extendability. Some old functions of the DailyRDAHMMThread class are also moved to the DailyRDAHMMStation class.

2. The process of modeling and evaluation for each station are changed to the following sequences:
(1) Check if a model already exists. If so, count the non-duplicated data in the model input -- if more than 200, skip the whole modeling process and go to step (5); otherwise delete the model files and continue with step (2).

(2) Choose intial start and end date for modeling input. First read the startDate saved in jplDates.txt or sopacDates.txt.
if startDate < 2006-09-30 and startDate + 1year <= 2006-09-30 then
endDate = 2006-09-30
else
if startDate + 1year <= "today" then
endDate = startDate + 1year
else
set endDate to null
give up the modeling process and return failure
endif
endif

(3) Adjust endDate.
while number of non-duplicated data in the model input of [startDate, endDate] is less than 200 do
If endDate > "today" then
set endDate to null
give up the modeling process and return failure
endif
endDate += 1year
endwhile

(4) Build the model.
Fill the model input with duplicated data, and build the model.

(5) Do evaluation.
Read the lastDate of model input from the model input file;
Query for input for [lastDate+1, "today"];
Append the queried input to the model input, and fill the combined input with duplicated data;
Do evaluation on the filled input.

3. Two sub-elements are added to each "station" element in the XML file of the station list: one "modelStartDate" sub-element and one "modelEndDate" sub-element. Since these dates are different for different context groups, we now have two different station list files for the two context groups (sopacGlobk and jplGipsy) analyzed.

4. The service is now scheduled to run every seven days, since we got informed that the GPS data are updated in a weekly manner. Maybe we should change the service name to "weekly RDAHMM service"?

Interactive plots added to daily RDAHMM portlet

We added interactive plots to the daily RDAHMM portlet at http://gw11.quarry.iu.teragrid.org/gridsphere/gridsphere?cid=DailyRDAHMM-Portlet. Now if you click on a station marker on the Google Map in the portlet, the pop-up window will show an interactive plot component, where you can view the station's GPS data plot in different time scales, from the most recent one week to all time.

The implementation of this interactive plot is an embedded flash object which does the plotting job given two links as arguments: one for the .raw file of the station's GPS data input, and one for the .Q file of the station's all-time state change sequence. This flash object is made by Sidd and Josh in PTI -- many thanks to their help with QuakeSim.

Some times the flash object could load slowly due to the network activity of the machine hosting it.