changes to daily rdahmm

Here are some important changes to daily rdahmm that we made these days but not posted here yet:

1. There are two phases for every station. The first phase is the training procedure, where rdahmm is run on the historical gps data (1994-01-01 to 2006-09-30) of every station to get a rdahmm model for this station. The second phase is to evaluate the daily data (data 2006-10-01 to "today") of every station with the model trained in first phase.

2. Portlet users can now select a specific date to see status change information of all stations on that day. A calendar and a slider bar is added to help the users choose the date. Besides, the colors used for map markers are now denoting different meanings. Red is used for markers of stations with a status change on the selected date, yellow is used for stations with status changes within 30 days before that date, and green is used for no status changes during that time. As for the output links, the model files (.A, .B, .pi, .range, .minval, .maxval, .L) are from the trained rdahmm models of the stations, while the evaluation files (.Q, .input, .png) are from the evaluation phase.

3. We now have a station list of all 460+ stations of California. However, the portlet is running very slow with such a large number of stations, because of the high workload of checking the status changes for all stations for every selected date, and rendering the markers on the map. I am trying to do modifications to both the format of the xml file of status changes and the checking algorithm to make it work.

some small modifications

1. We are temporarily using a fixed number, 7, as the number of states in running daily rdahmm detection;
2. Since "today's" results always contain the results of "yesterday", we only store the results of "today". The program automatically delete "yesterday's" resulting directories after completing "today's" detection;
3. The shrunk picture in the pop-up tabs of google markers are not clear, so we added an automatic selection function to the portlet. When the user clicks a marker, the drop-down list, status change table, and result link table on the right side will automatically switch to the station name corresponding to the clicked marker, so that the user can view the big pictures by clicking the links in the result link table.

Daily Rdahmm Detection Program

The program for daily detection and plots is now mostly done. This program runs as a stand alone process with a number of threads to run rdahmm on all stations concurrently. The number of threads can be specified as an argument to the program, with a default value of 8. Every thread schedules itself to run at 4:00am everyday, and the "number of states" parameter used for rdahmm is 5.

The program generates the resulting directories and plot images, as well as an xml file that contains both status change information within the last 10 days and the urls for the resulting files. Using this xml file, the portal can now show a list of all stations, and give the result urls for the station whose id is selected.

Implementation of the background service

A new class, DailyRDAHMMRunner, is added to Exec_RDAHMMService project, to run RDAHMM daily on all stations in background. This class contains a main() entry so that it can run as a stand alone process. This class creates a list of all stations with a state assigned to every station to denote if RDAHMM has been run “today” on this station. Initially every station’s state is assigned as waiting to be run. Then a specific number of threads (this number can be specified when the program is started) are created to run RDAHMM on a series of stations. Every station keeps trying to get the next waiting-to-be-run station from the list, runs RAHMM on it, and generates necessary results for the station. Once there are no waiting-to-be-run stations left in the list, the thread sleeps until the scheduled running time on “tomorrow”.

DailyRDAHMM portlet

I added a new "DailyRDAHMM" portlet to show the result of daily RDAHMM detection.

Upon the request of a browser, the portlet analyzes an xml file, which is generated after running daily rdahmm to record the status changes of stations, and shows the status changes in a similar way to the Realtime RDAHMM portlet. There is a mark for every station on the google map; a click on the mark shows the latest status change trace picture of that station. If the station has a status change from the day before to the present date, the corresponding mark has a red icon; if a status change is detected for a date in the history, the mark icon is yellow; green is used for no status change. Besides the map, a list containing all the stations with status changes and the date of the changes is shown on the right side of a web page.

A test example of this portlet can be accessed at http://156.56.104.131:8080/gridsphere/gridsphere?cid=DailyRDAHMM-Portlet . This example uses an arbitrary manually edited status-change xml file.

The codes for running rdahmm daily on all stations have not been completed, cause I got this problem of how to complete this daily schedule of invoking rdahmm services for all stations...I wonder if a standalone application should be added to call the service everyday. PS, since svn diff does not provide accurate information concerning the status changes and the date they happen, I think I'll just compare the files line by line, and just use svn to save the files.

a busy week

The page layout of daily RDAHMM detection will be similar to that of realtime RDAHMM, except that there will be an additional list to show all the stations for which state changes have been detected. So there is a lot to rely on in realtime RDAHMM. The invoking mode is also similar, just that daily detection needs a daily running schedule.

The daily detection portlet will always try to display the latest result. If the result for the current date is still not available, the previous day's result is given.

Got the os mid-term exam; not hard. Db mid-term is postponed, and this is not supposed to be a problem.

weird myeclipse

I got this weird problem with myeclipse that after installing it eclipse just turned out no responding. Then system files and directories began to disappear after I forced eclipse terminated, which resulted in system boot failure when the machine is restarted. I got the myeclipse from their official site... I don't know if there was any virus... anyway, fedora reinstalled.

I started to read some codes of RDAHMM. I think I can begin from the function for analysis of svn diff output before understanding the detailed flows.

The mid-term exams are coming next week, so probably I'll get little time with the project... But the courses are just getting more interesting, hehe.

Got the first problem to solve

Zhigang came on Wednesday, and we had a talk together with Marlon. They told me some details about the code and services in QuakeSim, and about the new function of running RDAHMM everyday on archived data.

I belive the implementation of this new function will take no long once I am familiar with both the technologies and the services and structures of QuakeSim. So I think I'll first do some code-reading of the services, and learn about web services technology on the way.

I am also looking forward to seeing interesting results of the new function, and learning something about the classification algorithms used by RDAHMM may also be fun.

Getting familiar with QuakeSim2

After solving a problem with downloading maven, I got QuakeSim2 built and installed.

I changed the google key in env.sh before building the project, but after that there are still several jsps that should be modified with the new google key. I don't know if there are other ways to do so, but things would be prettier if we can get this done automatically. I will talk about this with Zhigang when he comes.

I read two papers about QuakeSim. Nothing about bandwidth constraints in the development of QuakeSim is mentioned in these two papers. If QuakeSim is not facing this problem, the bandwidth bottle neck of the satellite channel should be one of the key differences between PolarGrid and QuakeSim.

not an easy start...

To get started with QuakeSim2, I first tried to install linux; but it turned out that things just don't go smoothly.

Firstly the partition magic software failed while I tried to merge f:\ driver to e:\ to set up space for linux, and the result was just as bad as getting f:\ lost. I tried to recover some files with easy recovery, and fortunately Qi told me that there has been nothing important under that drive.

Then the dvd-rom turned out unable to read the borrowed fedora disk.

So one thing led to another, I ended up installing fedora from iso image on vmware.

The operating system course is heavy, but I get a chance to get familiar with linux and svn from the experience with the assignments. So things should be easier after this first step.

Another problem is that I always get this "301 Moved" Error from svn when trying to check out the codes from source forge, either with the svn clinet on the servers of the cs department, or on the newly installed fedora, or in the windows environment. I did some search about this, and it seems that there might be some problems with the configuration of the server. I will talk to Malon about this.

God bless us all, and God bless QuakeSim.

Hello, my friends

I'm very happy to share my work and fun with all you guys, and thanks so much for the great help from everyone!