The gw11.quarry.iu.teragrid.org portal server and gf13.ucs.indiana.edu http server have been updated to use port 80 instead of 8080, so that access to the portal is not blocked by any firewalls. This update is completed by running httpd on the same servers and having httpd redirect requests to URL patterns corresponding to QuakeSim portal and resources to the QuakeSim tomcat servers. Therefore, the tomcat servers are still running on port 8080, and services which used to access the web services through 8080 can still work.
The realtime RDAHMM service configurations are modified so that the service now stores all results to directories under tomcat's "webapps" on gf13. The realtime RDAHMM portlet is also modified so that the results are now accessed from gf13 instead of gf14.
Thursday, October 8, 2009
XYZ changed to NEU in daily and realtime RDAHMM
The daily and realtime RDAHMM portlets have been updated so that the "Current X", "Current Y", "Current Z" tags are changed to "North", "East" and "Up".
Saturday, July 25, 2009
ATA over Ethernet V.S. iSCSI
I read the rfc specification of ATA over Ethernet (AoE) and a bunch of related documents about AoE and iSCSI, and got the following feature comparisons:
1. Routability. AoE is built directly on top of Ethernet, and thus can only be used inside LAN or VLAN, not over Internet -- it is practically impossible to build a VLAN over Internet. iSCSI can be deployed across the Internet because it is built on IP, and the initiator and target communicate through TCP connections.
2. Performance. AoE is more light weighted and has less network and CPU overhead than iSCSI since it runs directly over Ethernet, but a test done by VMWare shows that they can both reach a throughput at the wire speed under proper configurations. See http://www.vmware.com/files/pdf/storage_protocol_perf.pdf and http://www.coraid.com/site/co-pdfs/AoE_Performance_Comparison_3.pdf.
3. Sharing of devices, or targets. AoE specification states AoE as a connectionless protocol, and provides mechanisms such as reserve/release command and config string to coordinate the concurrent access from different hosts, but this is not a real target sharing mechanism among multiple hosts. For example, if one target is reserved by one specific host and that host goes down, there will be no normal way for another host to come over and resume the use of the target device. The only way to deal with this type of fail-over is to find the administrator to force release the target so that it is available to other hosts. Commercial products usually rely on higher level shared disk file systems to coordinate the access from multiple clients and handle node failures (see http://www.sourceteksystems.com/Uploads/AoE_Tutorial.pdf).On the contrary, iSCSI specification explicitly supports sharing of both devices and targets. That is, you can either export one device as multiple targets, or share one target among multiple initiators. So sharing of volumes is easier with iSCSI. For read-only sharing, we can simply have multiple instances attached to the same volume, either by creating multiple targets on the same physical or logical volume, or by connecting multiple initiators to a single device target. And in case of failure of one attached instance, we can directly have a backup instance come over and resume the use of the volume.
1. Routability. AoE is built directly on top of Ethernet, and thus can only be used inside LAN or VLAN, not over Internet -- it is practically impossible to build a VLAN over Internet. iSCSI can be deployed across the Internet because it is built on IP, and the initiator and target communicate through TCP connections.
2. Performance. AoE is more light weighted and has less network and CPU overhead than iSCSI since it runs directly over Ethernet, but a test done by VMWare shows that they can both reach a throughput at the wire speed under proper configurations. See http://www.vmware.com/files/pdf/storage_protocol_perf.pdf and http://www.coraid.com/site/co-pdfs/AoE_Performance_Comparison_3.pdf.
3. Sharing of devices, or targets. AoE specification states AoE as a connectionless protocol, and provides mechanisms such as reserve/release command and config string to coordinate the concurrent access from different hosts, but this is not a real target sharing mechanism among multiple hosts. For example, if one target is reserved by one specific host and that host goes down, there will be no normal way for another host to come over and resume the use of the target device. The only way to deal with this type of fail-over is to find the administrator to force release the target so that it is available to other hosts. Commercial products usually rely on higher level shared disk file systems to coordinate the access from multiple clients and handle node failures (see http://www.sourceteksystems.com/Uploads/AoE_Tutorial.pdf).On the contrary, iSCSI specification explicitly supports sharing of both devices and targets. That is, you can either export one device as multiple targets, or share one target among multiple initiators. So sharing of volumes is easier with iSCSI. For read-only sharing, we can simply have multiple instances attached to the same volume, either by creating multiple targets on the same physical or logical volume, or by connecting multiple initiators to a single device target. And in case of failure of one attached instance, we can directly have a backup instance come over and resume the use of the volume.
Friday, July 24, 2009
test version of VbsNimbusService deployed
We have got a basically working version of VBS integrated with Nimbus,
and I have deployed a test version of the integrated web service,
VbsNimbusService, to
http://cglc.uits.iu.edu:8080/axis2/services/VbsNimbusService?wsdl.
Some basic features include:
1) It is deployed under axis2, not globus; without any security
mechanisms or user management yet. So you can actually call it freely
with EPR http://cglc.uits.iu.edu:8080/axis2/services/VbsNimbusService/
.
2) It provides the same interface as EBS with SOAP binding. Note that
it accepts Nimbus workspace instance id numbers, such as 39 or 44, not
the handle names returned by the Nimbus cloud client.
3) The "progress" property of snapshot creation and "force" argument
of volume detach operation are not implemented yet. The snapshot
creation process is usually quick so you should be able to see the
status of a snapshot updated to "created" soon after the
createSnapshot operation returns. The "availabilityZone" property of a volume is meaningless in current VBS implementation, but we are keeping the information in the metadata database of the volumes.
4) For volumes newly created based on snapshots, there are 4 possible status: available, in-use, pending, and failed. "Available" means that the volume has been created successfully, and is available for attachment. "In-use" means that the volume is currently attached to some VM instance. "pending" means that the new volume is just being created by copying data from a given snapshot, and "failed" means that the volume has been created but the system didn't successfully copy data from the given snapshot to it. Note that in "pending" and "failed" status, the volume is actually also available for attachment; the status names are just an indication of information related to the snapshot-copy operation. It is not recommended to attach a "pending" volume to any VM instance, because operations from the instance may corrupt the copy process and lead to incorrect information written or read.
and I have deployed a test version of the integrated web service,
VbsNimbusService, to
http://cglc.uits.iu.edu:8080/axis2/services/VbsNimbusService?wsdl.
Some basic features include:
1) It is deployed under axis2, not globus; without any security
mechanisms or user management yet. So you can actually call it freely
with EPR http://cglc.uits.iu.edu:8080/axis2/services/VbsNimbusService/
.
2) It provides the same interface as EBS with SOAP binding. Note that
it accepts Nimbus workspace instance id numbers, such as 39 or 44, not
the handle names returned by the Nimbus cloud client.
3) The "progress" property of snapshot creation and "force" argument
of volume detach operation are not implemented yet. The snapshot
creation process is usually quick so you should be able to see the
status of a snapshot updated to "created" soon after the
createSnapshot operation returns. The "availabilityZone" property of a volume is meaningless in current VBS implementation, but we are keeping the information in the metadata database of the volumes.
4) For volumes newly created based on snapshots, there are 4 possible status: available, in-use, pending, and failed. "Available" means that the volume has been created successfully, and is available for attachment. "In-use" means that the volume is currently attached to some VM instance. "pending" means that the new volume is just being created by copying data from a given snapshot, and "failed" means that the volume has been created but the system didn't successfully copy data from the given snapshot to it. Note that in "pending" and "failed" status, the volume is actually also available for attachment; the status names are just an indication of information related to the snapshot-copy operation. It is not recommended to attach a "pending" volume to any VM instance, because operations from the instance may corrupt the copy process and lead to incorrect information written or read.
Two problems when writing the VbsNimbusService
Man, globus is not easy to use...
I encountered these two problems when writing the VbsNimbusService web service which integrates VBS with Nimbus:
1. We are trying to deserialize a workspace epr file in the execution of the service, but it reports a "no deserializer for {http://schemas.xmlsoap.org/ws/2004/03/addressing}AttributedURI" type of error when it comes to the XML type http://schemas.xmlsoap.org/ws/2004/03/addressing:AttributedURI. The problem is that the service found no type-mapping defined for this type. I did a lot of search and finally found a tip here: http://www.globus.org/toolkit/docs/4.0/common/javawscore/developer-index.html. There is the following notes about Java WS Core based client development:
"Any program that is based on Java WS Core should contain as a first entry in its classpath the directory of the Java WS Core installation. This is to ensure that the right client-config.wsdd is used by the client. That configuration file contains important client-side information such as handlers, type mappings, etc. "
That's it. There is a client-config.wsdd file under $GLOBUS_LOCATION which specifies deserialization classes for certain XML types. When you execute some clients deployed in globus, e.g., the Nimbus workspace client, their scripts make sure that the commands executing the clients' codes include $GLOBUS_LOCATION at the beginning of the CLASSPATH. But when you execute your own Java WS Core based clients, you need to add the directory containing that .wsdd file to the beginning of your CLASSPATH by yourself. And if that client is a web service itself, you will need to do this in the scripts that start your services' container. For example, if you deploy your service in tomcat, you will need to modify the catalina.sh file to specify the correct CLASSPATH.
2. The VbsNimbusService reports "Unkown CA" fault when doing authentication to the certificate of its own. VbsNimbusService uses a NimbusRPQueryClient to interact with the Nimbus workspace service. Like the Nimbus workspace client, NimbusRPQueryClient extends the org.globus.workspace.client_common.BaseClient class and thus also uses the credential got from GlobusCredential.getDefaultCredential() for authentication and authorization to the workspace service. The problem here is the trusted CA directory. For clients deployed in globus, the scripts running them specifies the trusted CA directory as the $X509_CERT_DIR environment variable, which contains the right trusted CAs. But when you are running your own Nimbus client without a globus container environment, the client's code will load the trusted CAs from some default directory. This "default directory" is just my guess, and I never even made it printed out so that I could copy the trusted CAs into it. Fortunately, I found the bug report here http://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=3843 and followed the instructions to set the trusted CA directory dynamically in the source codes of NimbusRPQueryClient, and finally made the VbsNimbusService work.
The instructions in the bug report above are not totally accurate. What we should actually do is to rewrite the setOptions(Stub stub) function in the class which extends org.globus.workspace.client_common.BaseClient (or org.globus.wsrf.client.BaseClient if you are not writing a Nimbus client), and add the following codes:
TrustedCertificates trustedCerts = TrustedCertificates.load("path to trusted certificates");
stub._setProperty(GSIConstants.TRUSTED_CERTIFICATES, trustedCerts);
super.setOptions(stub);
With this modification, your client should be able to set the right trusted CA directory without affecting the original execution of the BaseClient.
I encountered these two problems when writing the VbsNimbusService web service which integrates VBS with Nimbus:
1. We are trying to deserialize a workspace epr file in the execution of the service, but it reports a "no deserializer for {http://schemas.xmlsoap.org/ws/2004/03/addressing}AttributedURI" type of error when it comes to the XML type http://schemas.xmlsoap.org/ws/2004/03/addressing:AttributedURI. The problem is that the service found no type-mapping defined for this type. I did a lot of search and finally found a tip here: http://www.globus.org/toolkit/docs/4.0/common/javawscore/developer-index.html. There is the following notes about Java WS Core based client development:
"Any program that is based on Java WS Core should contain as a first entry in its classpath the directory of the Java WS Core installation. This is to ensure that the right client-config.wsdd is used by the client. That configuration file contains important client-side information such as handlers, type mappings, etc. "
That's it. There is a client-config.wsdd file under $GLOBUS_LOCATION which specifies deserialization classes for certain XML types. When you execute some clients deployed in globus, e.g., the Nimbus workspace client, their scripts make sure that the commands executing the clients' codes include $GLOBUS_LOCATION at the beginning of the CLASSPATH. But when you execute your own Java WS Core based clients, you need to add the directory containing that .wsdd file to the beginning of your CLASSPATH by yourself. And if that client is a web service itself, you will need to do this in the scripts that start your services' container. For example, if you deploy your service in tomcat, you will need to modify the catalina.sh file to specify the correct CLASSPATH.
2. The VbsNimbusService reports "Unkown CA" fault when doing authentication to the certificate of its own. VbsNimbusService uses a NimbusRPQueryClient to interact with the Nimbus workspace service. Like the Nimbus workspace client, NimbusRPQueryClient extends the org.globus.workspace.client_common.BaseClient class and thus also uses the credential got from GlobusCredential.getDefaultCredential() for authentication and authorization to the workspace service. The problem here is the trusted CA directory. For clients deployed in globus, the scripts running them specifies the trusted CA directory as the $X509_CERT_DIR environment variable, which contains the right trusted CAs. But when you are running your own Nimbus client without a globus container environment, the client's code will load the trusted CAs from some default directory. This "default directory" is just my guess, and I never even made it printed out so that I could copy the trusted CAs into it. Fortunately, I found the bug report here http://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=3843 and followed the instructions to set the trusted CA directory dynamically in the source codes of NimbusRPQueryClient, and finally made the VbsNimbusService work.
The instructions in the bug report above are not totally accurate. What we should actually do is to rewrite the setOptions(Stub stub) function in the class which extends org.globus.workspace.client_common.BaseClient (or org.globus.wsrf.client.BaseClient if you are not writing a Nimbus client), and add the following codes:
TrustedCertificates trustedCerts = TrustedCertificates.load("path to trusted certificates");
stub._setProperty(GSIConstants.TRUSTED_CERTIFICATES, trustedCerts);
super.setOptions(stub);
With this modification, your client should be able to set the right trusted CA directory without affecting the original execution of the BaseClient.
Tuesday, July 7, 2009
Three problems with the Nimbus testbed on cglc
We met three problems when trying to make Nimbus work on cglc.
1. The ethernet interface id on cglc5 starts from eth1 in stead of eth0. So the default network script setting of Xen will cause some strange effects on the network layout: no "peth"s, xenbr0 having no interfaces attached, etc. The solution is replace "(network-script network-bridge)" with "(network-script 'network-bridge netdev=eth1')" in the file /etc/xen/xend-config.sxp. cglc6 to cglc8 also have the same ethernet interface id sequences, and thus facing the same problem as cglc5. If any networking problems come up in the future on these machines, we should remember checking this configuration.
2. Ebtables of version numbers later than 2.0.6 have some bugs with Xen x86_64. Since ebtables are used by Nimbus, version numbers such as 2.0.8 or 2.0.9 will cause some exceptions with the following messages when new Nimbus workspaces are created:
"The kernel doesn't support a certain ebtables extension, consider recompiling your kernel or insmod the extension."
Accompanied by this dmesg error: "kernel msg: ebtables bug: please report to author: entries_size too small"
The solution is to build ebtables 2.0.6 on VMMs in Nimbus. A Nimbus user, Matt, has solved this before, and provided a rpm to the Nimbus site:
http://workspace.globus.org/downloads/ebtables-2.0.6-3.rf-mm-el5-x86_64.rpm
http://workspace.globus.org/downloads/ebtables-2.0.6-3.rf-mm-el5.src.rpm
sha1sums:
45982dacfaddfe8f828a720c94d7435a65a4bbc2 ebtables-2.0.6-3.rf-mm-el5-x86_64.rpm
df61bb2a43faa4fdc8e592c04b65242c71db790f ebtables-2.0.6-3.rf-mm-el5.src.rpm
And if you really want to build it manually, follow the steps bellow (this instruction is provided by Luna from the Nimbus user mail-group):
1) Down load the source codes of ebtables 2.0.6 from https://launchpad.net/ubuntu/dapper/+source/ebtables/2.0.6-3ubuntu2/+files/ebtables_2.0.6.orig.tar.gz;
2) Untar the source codes;
3) Down load the patches on http://svn.exactcode.de/t2/trunk/package/security/ebtables/, and apply them to the source codes;
4) Back up your previous ebtable installation (this is optional, just in case if you might need to use it in the future);
5) Do make and make install in the source directory of ebtables 2.0.6.
3. The default network association in worksp.conf provided by the Nimbus site is "association_0: private; xenbr0; vif0.0 ; none; 192.168.0.0/24", but this caused an exception on our testbed when I tried to deploy a new workspace: Nimbus is looking for an association named "public" instead of "private" when trying to configure the new virtual machine. So the solution is to replace "private" with "public" in this line. I don't know if this is general problem with the document on the Nimbus site http://workspace.globus.org/vm/TP2.2/admin/quickstart.html, but it works with our testbed. Maybe it's just because of some specific configuration on clgc5. Of course, you can always check the logs under /opt/workspace/logs to investigate the problems with workspace creation and termination.
1. The ethernet interface id on cglc5 starts from eth1 in stead of eth0. So the default network script setting of Xen will cause some strange effects on the network layout: no "peth"s, xenbr0 having no interfaces attached, etc. The solution is replace "(network-script network-bridge)" with "(network-script 'network-bridge netdev=eth1')" in the file /etc/xen/xend-config.sxp. cglc6 to cglc8 also have the same ethernet interface id sequences, and thus facing the same problem as cglc5. If any networking problems come up in the future on these machines, we should remember checking this configuration.
2. Ebtables of version numbers later than 2.0.6 have some bugs with Xen x86_64. Since ebtables are used by Nimbus, version numbers such as 2.0.8 or 2.0.9 will cause some exceptions with the following messages when new Nimbus workspaces are created:
"The kernel doesn't support a certain ebtables extension, consider recompiling your kernel or insmod the extension."
Accompanied by this dmesg error: "kernel msg: ebtables bug: please report to author: entries_size too small"
The solution is to build ebtables 2.0.6 on VMMs in Nimbus. A Nimbus user, Matt, has solved this before, and provided a rpm to the Nimbus site:
http://workspace.globus.org/downloads/ebtables-2.0.6-3.rf-mm-el5-x86_64.rpm
http://workspace.globus.org/downloads/ebtables-2.0.6-3.rf-mm-el5.src.rpm
sha1sums:
45982dacfaddfe8f828a720c94d7435a65a4bbc2 ebtables-2.0.6-3.rf-mm-el5-x86_64.rpm
df61bb2a43faa4fdc8e592c04b65242c71db790f ebtables-2.0.6-3.rf-mm-el5.src.rpm
And if you really want to build it manually, follow the steps bellow (this instruction is provided by Luna from the Nimbus user mail-group):
1) Down load the source codes of ebtables 2.0.6 from https://launchpad.net/ubuntu/dapper/+source/ebtables/2.0.6-3ubuntu2/+files/ebtables_2.0.6.orig.tar.gz;
2) Untar the source codes;
3) Down load the patches on http://svn.exactcode.de/t2/trunk/package/security/ebtables/, and apply them to the source codes;
4) Back up your previous ebtable installation (this is optional, just in case if you might need to use it in the future);
5) Do make and make install in the source directory of ebtables 2.0.6.
3. The default network association in worksp.conf provided by the Nimbus site is "association_0: private; xenbr0; vif0.0 ; none; 192.168.0.0/24", but this caused an exception on our testbed when I tried to deploy a new workspace: Nimbus is looking for an association named "public" instead of "private" when trying to configure the new virtual machine. So the solution is to replace "private" with "public" in this line. I don't know if this is general problem with the document on the Nimbus site http://workspace.globus.org/vm/TP2.2/admin/quickstart.html, but it works with our testbed. Maybe it's just because of some specific configuration on clgc5. Of course, you can always check the logs under /opt/workspace/logs to investigate the problems with workspace creation and termination.
Friday, June 19, 2009
VBS Architecture
The job of VBS is done on two ends: the volume server, the Xen manager, and through the interactions between the two. So our architecture is composed of three components: the VBS web service, the volume server delegate web service, and the VMM delegate web service.
The VBS web service is the front service that answers the requests from VBS clients, and coordinates the operations on the volume server and Xen VMMs. It provides the following method interfaces to the VBS client:
- create-volume
- delete-volume
- describe-volumes
- describe-volume
- create-snapshot
- delete-snapshot
- describe-snapshots
- describe-snapshot
- attach-volume
- detach-volume
The volume sever delegate web service is the service sitting on the logical volume server and encapsulating the LVM and ISCSI command lines, which are necessary to complete VBS operations. It is called by the VBS web service, and completes the operations of creating/deleting volumes/snapshots, exporting/deleting ISCSI targets by executing LVM and ISCSI commands.
The VMM delegate web service is the service sitting on VMM nodes and encapsulating VMM and ISCSI command lines. It is called by the VBS web service, and completes the operations of discovering and importing ISCSI targets, and attaching and detaching the ISCSI device to/from Xen DomU VMs.
The work flow of VBS is controlled by the VBS web service to complete the functions. A typical work flow for the attach-volume operation is composed of the following invocations carried out by the VBS web service:
(1) call the volume server delegate service to export an ISCSI target for;
(2) call the VMM delegate service on to discover the new ISCSI target, and log in to it. This will result in a new ISCSI device created on the VMM node;
(3) call the VMM delegate service to attach the corresponding ISCSI device to, which will be visible as inside the VM;
(4) update the metadata about the, so that consequent calls to describe-volume(s) about this volume will return consistent information.
The VBS web service is the front service that answers the requests from VBS clients, and coordinates the operations on the volume server and Xen VMMs. It provides the following method interfaces to the VBS client:
- create-volume
- delete-volume
- describe-volumes
- describe-volume
- create-snapshot
- delete-snapshot
- describe-snapshots
- describe-snapshot
- attach-volume
- detach-volume
The volume sever delegate web service is the service sitting on the logical volume server and encapsulating the LVM and ISCSI command lines, which are necessary to complete VBS operations. It is called by the VBS web service, and completes the operations of creating/deleting volumes/snapshots, exporting/deleting ISCSI targets by executing LVM and ISCSI commands.
The VMM delegate web service is the service sitting on VMM nodes and encapsulating VMM and ISCSI command lines. It is called by the VBS web service, and completes the operations of discovering and importing ISCSI targets, and attaching and detaching the ISCSI device to/from Xen DomU VMs.
The work flow of VBS is controlled by the VBS web service to complete the functions. A typical work flow for the attach-volume operation is composed of the following invocations carried out by the VBS web service:
(1) call the volume server delegate service to export an ISCSI target for
(2) call the VMM delegate service on
(3) call the VMM delegate service to attach the corresponding ISCSI device to
(4) update the metadata about the
Manual scripts for performing the functions of VBS
We'll have a series of documents explaining the design and implementation of the Virtual Block Store (VBS) system.
This post presents the basic manual steps to complete the VBS functions: create a new logical volume, attach it to some Xen DomU VM, detach it from the VM, create a snapshot of it, delete the snapshot, and delete the volume. The manual scripts are based on existing softwares, including LVM (Logical Volumen Manager), ISCSI, and Xen.
Test steps for creating, attaching and detaching volumes:
1) create a new volume on the volume server:
lvcreate -n gxmVol01 -L32M /dev/vg
2) export a new iscsi target:on the volume server
ietadm --op new --tid=2 --params Name=iqn.2009-03.edu.iu.cgl:cglch.lun0
ietadm --op new --tid=2 --lun=0 --params Path=/dev/vg/gxmVol01
3) discover and login the new iscsi target from a Xen dom0:
iscsiadm --mode discovery --type sendtargets --portal 192.168.1.254:3260
iscsiadm --mode node --targetname iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --login
After log-in, check the disk directory for the newly created ISCSI device: ls /dev/disk/by-path
4) do block-attach in Xen dom0 to attach the iscsi block device to a domU:
xm block-attach ttylinux phy:/dev/disk/by-path/ip-192.168.1.254:3260-iscsi-iqn.2009-03.edu.iu.cgl:cglch.lun0-lun-0 /dev/sdb w
5) to detach a volume from a domU, do xm block-detach in dom0:
xm block-detach ttylinux /dev/sdb
Note: don't do "xm block-attach" while the block device is mounted on the corresponding domU VM. If there is no xenconsole connected to the VM when the block-detach happens, you will get an error the next time you try to connect to the VM through xenconsole, and it's probable that you'll never be able to login to the VM again by xenconsole. So always remember to do umount before block-detach.
6) logout from the iscsi target in dom0:
iscsiadm --mode node --targetname iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --logout
7) delete the iscsi target from the volume server:
ietadm --op delete --tid=2
8) create a snapshot of a volume:
lvcreate --size 32m --snapshot --name gxmSnap01 /dev/vg/gxmVol1
9) delete the volume or snapshot:
lvremove /dev/vg/gxmVol1
This post presents the basic manual steps to complete the VBS functions: create a new logical volume, attach it to some Xen DomU VM, detach it from the VM, create a snapshot of it, delete the snapshot, and delete the volume. The manual scripts are based on existing softwares, including LVM (Logical Volumen Manager), ISCSI, and Xen.
Test steps for creating, attaching and detaching volumes:
1) create a new volume on the volume server:
lvcreate -n gxmVol01 -L32M /dev/vg
2) export a new iscsi target:on the volume server
ietadm --op new --tid=2 --params Name=iqn.2009-03.edu.iu.cgl:cglch.lun0
ietadm --op new --tid=2 --lun=0 --params Path=/dev/vg/gxmVol01
3) discover and login the new iscsi target from a Xen dom0:
iscsiadm --mode discovery --type sendtargets --portal 192.168.1.254:3260
iscsiadm --mode node --targetname iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --login
After log-in, check the disk directory for the newly created ISCSI device: ls /dev/disk/by-path
4) do block-attach in Xen dom0 to attach the iscsi block device to a domU:
xm block-attach ttylinux phy:/dev/disk/by-path/ip-192.168.1.254:3260-iscsi-iqn.2009-03.edu.iu.cgl:cglch.lun0-lun-0 /dev/sdb w
5) to detach a volume from a domU, do xm block-detach in dom0:
xm block-detach ttylinux /dev/sdb
Note: don't do "xm block-attach" while the block device is mounted on the corresponding domU VM. If there is no xenconsole connected to the VM when the block-detach happens, you will get an error the next time you try to connect to the VM through xenconsole, and it's probable that you'll never be able to login to the VM again by xenconsole. So always remember to do umount before block-detach.
6) logout from the iscsi target in dom0:
iscsiadm --mode node --targetname iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --logout
7) delete the iscsi target from the volume server:
ietadm --op delete --tid=2
8) create a snapshot of a volume:
lvcreate --size 32m --snapshot --name gxmSnap01 /dev/vg/gxmVol1
9) delete the volume or snapshot:
lvremove /dev/vg/gxmVol1
Sunday, April 12, 2009
Modifications to daily RDAHMM service for supporting JPL data source
We have modified the daily RDAHMM service in the following way so that other data sources can be supported in a flexible way:
1. We run one service process for each data source;
2. Corresponding to different data sources, different directories will be used to save the daily RDAHMM analysis results. Besides, different arguments should be used for querying the input GPS data for different data sources;
3. On start up, a service process will receive the path to a property file as an argument. This file contains information such as the data source, pre-processing treatment, directory for saving the results, and GRWS query arguments for this process. The process will analyze the property file, and thus do analysis to the specified data source, using the right corresponding query arguments, pre-processing treatment, and output directory.
1. We run one service process for each data source;
2. Corresponding to different data sources, different directories will be used to save the daily RDAHMM analysis results. Besides, different arguments should be used for querying the input GPS data for different data sources;
3. On start up, a service process will receive the path to a property file as an argument. This file contains information such as the data source, pre-processing treatment, directory for saving the results, and GRWS query arguments for this process. The process will analyze the property file, and thus do analysis to the specified data source, using the right corresponding query arguments, pre-processing treatment, and output directory.
Support for JPL data source added to Daily RDAHMM portlet
Support for JPL data source has been added to daily RDAHMM portlet. We added a separate jsp page, DailyRdahmm_jpl.jsp, to show the results of JPL data source, and added a navigation rule in the portal configuration file, faces-conf.xml, to navigate betweent the pages for SOPAC data source and JPL data source. Now there is a link on each page pointing to the other page, so the user can click this link to view results for the other data source. Updates have been applied to the server at gw11.quarry.iu.teragrid.org:
http://gw11.quarry.iu.teragrid.org:8080/gridsphere/gridsphere
http://gw11.quarry.iu.teragrid.org:8080/gridsphere/gridsphere
Thursday, March 26, 2009
Commands needed for EBS operations
I found that the scripts for creating, attach and detach volumes in our Nimbus EBS test bed are not totally correct, so I looked up all the relevant documents and got the following steps for doing all these stuff:
1) create a new volume on the volume server::
lvcreate -n gxmVol01 -L32M /dev/vg
2) export a new iscsi target:
ietadm --op new --tid=2 --params Name=iqn.2009-03.edu.iu.cgl:cglch.lun0
ietadm --op new --tid=2 --lun=0 --params Path=/dev/vg/gxmVol01
3) discover and login the new iscsi target from a Xen dom0:
ssh root@cglc5 iscsiadm --mode discovery --type sendtargets --portal 192.168.1.254:3260
ssh root@cglc5 iscsiadm --mode node --targetname iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --login
4) do block-attach in Xen dom0 to attach the iscsi block device to a domU:
ssh root@cglc5 xm block-attach ttylinux phy:/dev/disk/by-path/ip-192.168.1.254:3260-iscsi-iqn.2009-03.edu.iu.cgl:cglch.lun0-lun-0 /dev/sdb w
After step 4), you will be able to create new partitions and file systems on the front end device /dev/sdb, and mount that file system to some directory. Note that the iscsi target should be attached to a scsi device symbol.
5) to detach a volume from a domU, do xm block-detach in dom0:
ssh root@cglc5 xm block-detach ttylinux /dev/sdb
And of course, if you have a mounted file system on this device, you need to umount it first before doing this.
6) logout from the iscsi target in dom0:
ssh root@cglc5 iscsiadm --mode node --targetname
iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --logout
Next I'll try to wrap these commands with web services and build a prototype of EBS.
1) create a new volume on the volume server::
lvcreate -n gxmVol01 -L32M /dev/vg
2) export a new iscsi target:
ietadm --op new --tid=2 --params Name=iqn.2009-03.edu.iu.cgl:cglch.lun0
ietadm --op new --tid=2 --lun=0 --params Path=/dev/vg/gxmVol01
3) discover and login the new iscsi target from a Xen dom0:
ssh root@cglc5 iscsiadm --mode discovery --type sendtargets --portal 192.168.1.254:3260
ssh root@cglc5 iscsiadm --mode node --targetname iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --login
4) do block-attach in Xen dom0 to attach the iscsi block device to a domU:
ssh root@cglc5 xm block-attach ttylinux phy:/dev/disk/by-path/ip-192.168.1.254:3260-iscsi-iqn.2009-03.edu.iu.cgl:cglch.lun0-lun-0 /dev/sdb w
After step 4), you will be able to create new partitions and file systems on the front end device /dev/sdb, and mount that file system to some directory. Note that the iscsi target should be attached to a scsi device symbol.
5) to detach a volume from a domU, do xm block-detach in dom0:
ssh root@cglc5 xm block-detach ttylinux /dev/sdb
And of course, if you have a mounted file system on this device, you need to umount it first before doing this.
6) logout from the iscsi target in dom0:
ssh root@cglc5 iscsiadm --mode node --targetname
iqn.2009-03.edu.iu.cgl:cglch.lun0 --portal 192.168.1.254:3260 --logout
Next I'll try to wrap these commands with web services and build a prototype of EBS.
Monday, March 23, 2009
Daily RDAHMM portlet modified corresponding to the video service
We have modified the daily RDAHMM portlet so that it now links to a video file made by the daily RDAHMM video service, which will always be in mp4 format.
Monday, March 16, 2009
Daily RDAHMM Video Web Service Completed
And here's the design document:
To make daily RDAHMM movies with h264 video format, which is the most portable format across Linux, Windows and Mac platforms, we decide to create a DailyRDAHMMVideoService web service sitting on a machine where everything is OK for making h264 videos, and answering the requests from DailyRDAHMMService, and returning links to videos newly created.
Pre-requisite:
mencoder installed with support for x264 options
mp4BOX installed
So far the only machine that satisfies these requirements is Xiaoming's machine with IP address 156.56.104.161.
Requirements for DailyRDAHMMVideoService web service:
We call DailyRDAHMMVideoService the "video service" for short. This service is supposed to support the creation of daily RDAHMM movies for different data sources (SOPAC, JPL, etc.), and different pre-process treatment methods to input data (with vs. without missing data treatment). Given a data source, a treatment method, a URL to the analysis result XML file of that data souce, and a pair of start and end date, the video service will create a movie for the period between the start and end date, according to the result XML file specified in that URL, and return a URL to the created video. More specificly, the video service provides a web service method "String makeVideo(String dataSource, String preProcTreat, String resultURL, String startDate, String endDate)", which makes the requested video, and returns the URL string of the video.
Design Issues:
1. Directory seperation. Since we want to support different data sets and input treatment methods, the analysis results and corresponding map pictures for these different parameters will be different, so we need to separate the directories for different parameter combinations. Let denote the top directory for storing the pictures and temporary video files, we use [picDir]/[dataSouce]/[preProcTreat] for different parameters.
2. Quick response. To respond quickly to the client, we apply asynchronous process to its request. First, the video service will check if the parameters are valid, e.g., whether the given data source and treatment methods are supported. If not, a URL pointing to another method of the video service, "String getErrorMsg(int msgNum)" will be returned to the client, so that when the client opens that URL, a proper error message will be returned. If the parameters are valid, a URL will be constructed immediately according to these parameters and returned to the client, before the video is actually created. That is, we assume that there will be no errors in the process of creating the video. After enough debugging, we think this will not cause very serious problems: anyway, putting an invalid URL on the portal in rare cases will not be the end of the world.
3. Requests buffering. In order to answer multiple requests asynchronously in a short time, we buffer these results in a list, and use a separate thread to process them. This separate thread is sleeping when there is no request to process, and will be waken up for new requests.
4. Avoiding redundant work. To avoid creating video frames for any period for multiple times, we save the start date (which will always be 1994-01-01) and end date of the last video created. Upon every new request, the video service will check its start and end date. If these dates falls in between the start and end date of the last video, we just return the URL for the last video. A new video will be created only if the end date is later than the end date of the last video, since all starting dates are supposed to be no earlier than 1994-01-01. Besides, if the month number of the given end date minus 2 is later than the month number of the historical video, the historical video is updated and extended to the month according to the given month number minus 2. For example, if the end date of the historical video is 2008-07-31, and the end date of the requested video is 2008-11-22, then the historical video will be updated to 2008-09-30 first, and then the new video is created based on the new historical video.
To make daily RDAHMM movies with h264 video format, which is the most portable format across Linux, Windows and Mac platforms, we decide to create a DailyRDAHMMVideoService web service sitting on a machine where everything is OK for making h264 videos, and answering the requests from DailyRDAHMMService, and returning links to videos newly created.
Pre-requisite:
mencoder installed with support for x264 options
mp4BOX installed
So far the only machine that satisfies these requirements is Xiaoming's machine with IP address 156.56.104.161.
Requirements for DailyRDAHMMVideoService web service:
We call DailyRDAHMMVideoService the "video service" for short. This service is supposed to support the creation of daily RDAHMM movies for different data sources (SOPAC, JPL, etc.), and different pre-process treatment methods to input data (with vs. without missing data treatment). Given a data source, a treatment method, a URL to the analysis result XML file of that data souce, and a pair of start and end date, the video service will create a movie for the period between the start and end date, according to the result XML file specified in that URL, and return a URL to the created video. More specificly, the video service provides a web service method "String makeVideo(String dataSource, String preProcTreat, String resultURL, String startDate, String endDate)", which makes the requested video, and returns the URL string of the video.
Design Issues:
1. Directory seperation. Since we want to support different data sets and input treatment methods, the analysis results and corresponding map pictures for these different parameters will be different, so we need to separate the directories for different parameter combinations. Let
2. Quick response. To respond quickly to the client, we apply asynchronous process to its request. First, the video service will check if the parameters are valid, e.g., whether the given data source and treatment methods are supported. If not, a URL pointing to another method of the video service, "String getErrorMsg(int msgNum)" will be returned to the client, so that when the client opens that URL, a proper error message will be returned. If the parameters are valid, a URL will be constructed immediately according to these parameters and returned to the client, before the video is actually created. That is, we assume that there will be no errors in the process of creating the video. After enough debugging, we think this will not cause very serious problems: anyway, putting an invalid URL on the portal in rare cases will not be the end of the world.
3. Requests buffering. In order to answer multiple requests asynchronously in a short time, we buffer these results in a list, and use a separate thread to process them. This separate thread is sleeping when there is no request to process, and will be waken up for new requests.
4. Avoiding redundant work. To avoid creating video frames for any period for multiple times, we save the start date (which will always be 1994-01-01) and end date of the last video created. Upon every new request, the video service will check its start and end date. If these dates falls in between the start and end date of the last video, we just return the URL for the last video. A new video will be created only if the end date is later than the end date of the last video, since all starting dates are supposed to be no earlier than 1994-01-01. Besides, if the month number of the given end date minus 2 is later than the month number of the historical video, the historical video is updated and extended to the month according to the given month number minus 2. For example, if the end date of the historical video is 2008-07-31, and the end date of the requested video is 2008-11-22, then the historical video will be updated to 2008-09-30 first, and then the new video is created based on the new historical video.
Wednesday, February 11, 2009
Compilation problem with x264 enabled mencoder
In order to make a daily RDAHMM movie with fps=5 that is compatible to every platform and player, we tried to encode the video with several different types of formats, and found that the only qualified format is h264. However, in order to make mencoder work with h264, we need to recompile it with x264 option, as well as the support of x264 package. We found it very hard to make sure the compilation can complete on any server, because of lack of x264 support on the machine, or because the wrong version of x264.
So we are now trying to separate the video making function from the Daily RDAHMM Service, so that it can just be deployed to a server where everything is OK, and answer requests from Daily RDAHMM Service from everywhere. This way we also need only one copy of all the map pictures and historical video, and the update of the historical video will also be easier to manage.
So we are now trying to separate the video making function from the Daily RDAHMM Service, so that it can just be deployed to a server where everything is OK, and answer requests from Daily RDAHMM Service from everywhere. This way we also need only one copy of all the map pictures and historical video, and the update of the historical video will also be easier to manage.
Tuesday, February 10, 2009
Daily RDAHMM Result Service added to svn control
The Daily RDAHMM Service has been added to the code depository of QuakeSim2. A new directory DailyRDAHMMResultService is added to the ExecutionServices directory, and the service can be built and deployed with maven commands from the top QuakeSim2 directory.
Since Daily RDAHMM Result Service is deployed with Axis2, we also added a axis2.tar.gz under the portal_deploy directory, which will be deployed after tomcat is deployed. The Daily RDAHMM Result Service is deployed under the "services" directory of axis2.
For sake of consistency with previous deployment configurations, the Daily RDAHMM Result Service will also be built with maven scripts in pom.xml, and a directory will be copied to the webapps directory of tomcat. But this is not necessary: this directory will not be the running service. We might be able to deploy all other services to axis2 in the future, so that we don't need to copy them to "webapps" directly any more.
Since Daily RDAHMM Result Service is deployed with Axis2, we also added a axis2.tar.gz under the portal_deploy directory, which will be deployed after tomcat is deployed. The Daily RDAHMM Result Service is deployed under the "services" directory of axis2.
For sake of consistency with previous deployment configurations, the Daily RDAHMM Result Service will also be built with maven scripts in pom.xml, and a directory will be copied to the webapps directory of tomcat. But this is not necessary: this directory will not be the running service. We might be able to deploy all other services to axis2 in the future, so that we don't need to copy them to "webapps" directly any more.
Subscribe to:
Comments (Atom)
