weekly report 20121004

Done in last week:

Solved the following issues with the MyHBase setup:

1. An unsynced-clock exception on a region server caused by its out-of-sync system clock. A fuguregrid administrator helped solve this issue.

2. A timeout exception from Hadoop data node. This was fixed by setting the timeout value to 0 in Hadoop configuration files.

3. A "block-missing" exception from Hadoop data node. This exception was gone after the first two issues were resolved.

4. A "path not found" exception from Hadoop task tracker. After some investigation, I found that this might just be normal some status report from the task tracker that does not necessarily indicate an error. So we will try to live with it for now.

5. A crashing problem with an HBase region server in some runs. This might have been caused by the previous issues. After solving those issues and adjusting the memory limit of data nodes and region servers, the problem seems gone. Will investigate it again if it still happens.

6. A "socket connection refused" exception from the zookeepers. This seemed to happen during the start time of the servers, maybe because the servers were not totally started yet. I have put a 30-second wait time between the zookeeper start-up and the HBase start-up so that the zookeepers get a time to make everything right. We need more test runs to verify if this works.

To do next:

Start running on larger data scale.