Home -> VM -> TP1.2.2 -> Admin Guide / User Guide

Workspace Admin Guide

1. Introduction

This guide contains configuration information for system administrators working with Workspaces. It provides references to information on procedures typically performed by system administrators, including installing, configuring, deploying, and testing the installation.

Important

This information is in addition to the basic Globus Toolkit prerequisite, overview, installation, security configuration instructions in the GT 4.0 System Administrator's Guide. Please read through this guide before continuing.

Configuring GT4 with the bare necessities for the Workspace Service requires you to install the Java GT4 core with security tools:

make wsjava common globus_proxy_utils globus_simple_ca \
globus_simple_ca_setup postinstall

If you can obtain certificates and proxy tools from some other location, all that is required is Java GT4 core which is available as a binary tarball from the Globus download page.

Important

If you only need to install the workspace client see the client only section of this manual before proceeding. Some of the client-only installation options do not require you to install GT4 first.

2. Installing just the workspace client

There are different ways to install a workspace client environment:

Once installed, see the Workspace User Guide.

2.1. Using a preloaded Xen VM with client environment

  1. Download the Xen based client image, listed on the Workspace download page.

  2. Untar the archive and edit the enclosed config file to point at the correct kernel and image.

  3. Log in as root (password is 'root'). As root, run this:

    su - globus

    to log in to the Globus user environment.

For more information about the client environment on this VM (and in general), see the Workspace User Guide.

The VM is set up to DHCP, you can change this by editing the /etc/conf.d/net file. The file contains a commented out example of a static IP assignment. Then run

/etc/init.d/net.eth0 restart

2.2. Using a preloaded VMware VM with client environment

  1. Download the VMware based client image, listed on the Workspace download page.

  2. Untar the archive and point the VMware GUI to the enclosed vmx file (File -> Open).

  3. Log in as root (password is 'root'). As root, run this:

    su - globus

    to log in to the Globus user environment.

For more information about the environment on this VM (and in general about the client environment), see the Workspace User Guide.

The VM is set up to DHCP, you can change this by editing the /etc/conf.d/net file. The file contains a commented out example of a static IP assignment. Then run

/etc/init.d/net.eth0 restart

. The Workspace User Guide discusses the exact steps to take to accomplish this in the VM Networking section.

2.3. Installing the reference client and Globus runtime

The basic Globus Java client environment almost entirely overlaps with the server environment. We provide an option to download a tarball of this environment with the Workspace sample client already installed into it.

Installing the client this way assumes you have a working Java JRE (1.4+) as well as externally obtained certificates.

The proxy generation tools included with this tarball are not the recommended tools for use with Linux. There is a problem with Java's interaction with Linux terminals that results in typed characters possibly being seen by an onlooker.

  1. Download the "Binary, Globus Java core plus client" tarball from the Workspace download page.

  2. Expand the tarball and set your GLOBUS_LOCATION environment variable to the created directory.

  3. Now see the Workspace User Guide.

2.4. Installing the reference client from binary GAR files

The GAR format is a portable, binary archive of everything needed to deploy the client into a pre-existing container.

Installing the client via GAR files assumes you have a working GT4 Java core installation. A Java JRE (1.4+) and Ant (1.6+ and 1.6.1+ if using Java 1.5) is assumed.

Configuring GT4 with the bare necessities for the Workspace reference client requires you to install the Java GT4 core with security tools:

make wsjava common globus_proxy_utils globus_simple_ca \
globus_simple_ca_setup postinstall

If you can obtain certificates and proxy tools from some other location, all that is required is Java GT4 core which is available as a binary tarball from the Globus Toolkit download page.

  1. Download the "Client GAR files" tarball from the Workspace download page and expand it to a random directory..

  2. Set the GLOBUS_LOCATION environment variable as your environment requires (bourne shell shown):

    export GLOBUS_LOCATION=/path/to/globus

  3. Set the ANT_HOME environment variable as your environment requires (bourne shell shown):

    export ANT_HOME=/path/to/ant/home

  4. Deploy the three GAR files:

    cd $GLOBUS_LOCATION
    ./bin/globus-deploy-gar workspace_schemas.gar
    ./bin/globus-deploy-gar workspace_common.gar
    ./bin/globus-deploy-gar workspace_client.gar

    More detailed instructions are available for globus-deploy-gar and globus-undeploy-gar if necessary.

  5. Now see the Workspace User Guide.

2.5. Installing the reference client from source

Installing the client from source assumes you have a working GT4 Java core installation. A Java JDK (1.4+) and Ant (1.6+ and 1.6.1+ if using Java 1.5) is assumed.

Configuring GT4 with the bare necessities for the Workspace reference client requires you to install the Java GT4 core with security tools:

make wsjava common globus_proxy_utils globus_simple_ca \
globus_simple_ca_setup postinstall

If you can obtain certificates and proxy tools from some other location, all that is required is Java GT4 core which is available as a binary tarball from the Globus Toolkit download page.

  1. Download the "Grid Service" source tarball from the Workspace download page.

  2. Set the GLOBUS_LOCATION environment variable as your environment requires (bourne shell shown):

    export GLOBUS_LOCATION=/path/to/globus

  3. Set the ANT_HOME environment variable as your environment requires (bourne shell shown):

    export ANT_HOME=/path/to/ant/home

  4. The root build file for the Workspace Service is workspace-service/build.xml and it contains a special build target to just build the client.

    Build and install the client:

    cd workspace-service
    ant client

  5. Now see the Workspace User Guide.

3. Installing the Workspace service and backend from source

Download the grid service and workspace-control from the download page.

One tarball contains the GT4 services and the other contains the workspace-control program which is required to be installed on a physical node running the Xen hypervisor. The service and workspace-control can be installed on the same node or on different nodes. And further, you can install the workspace-control program to multiple hypervisors and the service will manage them as a pool of resources.

3.1. Installing the Workspace Service.

Currently, only the source is distributed for the service. The root build file for the Workspace Service is workspace-service/build.xml

This will install the client and service:

export GLOBUS_LOCATION=/path/to/GL
cd workspace-service
ant deploy

Before testing the service, you need to configure it and install and configure the workspace-control program.

As of this TP1.2.2 version of the workspace service you also need to install DHCP and ebtables on each hypervisor node. For more information on why this is now necessary and how it will not interfere with a site's pre-existing DHCP server, see the DHCP overview and configuration section.

3.2. Installing the workspace-control program.

The program is implemented in Python and comes with its own installer. It requires Python 2.3+

If you are planning on installing workspace-control to multiple hypervisor nodes so that the service can manage the nodes as a resource pool, you should install to one first. Once you have edited the configuration file as you need to, it can all be replicated across your resource pool by using the automatic installer.

For Xen3, network information delivery to workspaces, and for any functionality requiring workspace image editing (currently only certificate staging), the sudo program must be installed on the same node as the backend program. See the configuring sudo section for more information.

For workspace-control to support networking information delivery to VMs, you are required to install DHCP and ebtables on each hypervisor node. When networking information in a workspace needs to change at its startup (which is typical upon deployment), workspace-control will make a call via sudo to a program that adds a MAC address to IP mapping into the local DHCP server for each of the workspace's NICs that need to be configured. It will also adjust ebtables rules for each of the workspace's NICs that need to be configured: these make sure the NICs are using the proper MAC and IP address as well as directing DHCP requests to the local DHCP server only. For more information on why this is now necessary and how it will not interfere with a site's pre-existing DHCP server, see the DHCP overview and configuration section.

The install.py program has a help switch (-h) that will explain its options, but briefly there are three main modes of the installer, excert from the -h output:

  -n, --noninteractive  Don't ask the user anything (for automated install with
                        a well-known conf file).

  -o, --onlyverify      Just run the setup tests and print what would have
                        happened in --noninteractive mode.  Good option to try
                        first since it will do nothing to the filesystem.

  -i, --install         Install the program, make needed directories, and set
                        them with default permissions. Will block to ask you
                        questions if necessary.

The --noninteractive mode is a good, fast choice if you have no /opt/workspace directories on your system and just want to use the default /opt/workspace hierarcy that comes in the sample configuration file. The installer makes adding hypervisor nodes to your resource pool very easy, once you have decided on some of the configuration choices.

For more on running the installer, see the configuration section.

3.3. Installing DHCP and ebtables

For an overview of what these tools are used for and how they are situated in terms of your site's network setup, see the DHCP overview and configuration section.

For a lot of hypervisor nodes this may not be necessary as the packages are already present. Check your distribution tools for ISC's DHCP server and ebtables. Or simply try to run the command "/sbin/ebtables" and look for the "/etc/dhcp/dhcpd.conf" file and accompanying "/etc/init.d/dhcpd" script. The ISC DHCP server is the most standard and widely supported DHCP server so it is likely that is what you have installed.

All major Linux distributions include these two packages, you should be able to easily install them with your package management system. For example, "rpm -ihv dhcp-*.rpm", "apt-get install dhcp", "emerge dhcp", etc. And the same for ebtables.

ebtables requires kernel support in dom0, the default Xen kernel includes this support. If your dom0 kernel does not include these for some reason, the options to enable are under Networking :: Networking options :: Network packet filtering :: Bridge Netfilter Configuration :: Ethernet Bridge Tables

Any recent version of each package should be compatible, the distributed scripts that automate the configurations were tested with ISC DHCP 3.0.3 and ebtables 2.0.6 userspace tools.

The scripts that interact with the packages are straightforward and can be swapped with alternate implementations if you need to; also feel free to ask questions about the implementation on workspace-user@globus.org.

4. Configuring

4.1. Workspace Service configuration

After installing the Workspace Service, files that alter its configuration are stored in the $GLOBUS_LOCATION/etc/workspace_service/ directory.

4.1.1. Factory grid-mapfile

To edit the list of authorized users of the factory service, add DNs to $GLOBUS_LOCATION/etc/workspace_service/workspace-grid-mapfile

Important

The factory service grid-mapfile overrides the container wide grid-mapfile (typically this is /etc/grid-security/grid-mapfile). The service is distributed with zero entries, which disables access to workspace creation entirely.

Add a DN to the list in normal Globus fashion (if the DN has spaces in it, use quotes around it). The gridmap authorization module requires a username mapping, but it is irrelevant to this service, so using e.g. "fakeuser" for each DN entry is OK. We support grid-mapfiles instead of simple access control lists for maximum compatibility with predeployed installations.

Other (attribute based) authorization methods are now available via the plugin system.

After a workspace is created, only the creator may manage the WSRF resource. This is configured programmatically for each workspace, so no grid-mapfile for the workspace service is needed, just the factory.

4.1.2. Service implementation

There are currently two service implementations, xenlocal and xenSSH.

xenlocal does not handle the resource pool model, it is only for the cases where you want to run one workspace service on the same node as the (sole) hypervisor it manages.

xenSSH on the other hand will enable you to manage a pool of resources from one running service. The service keeps track of the what resource are available at the current time, adding and subtracting the resources used, keeping a view of what is available at the different hypervisor nodes. In this version of the service, the first introduction of the resource pool model, this only tracks the RAM available for VMs at the node. To configure the resource pools, see here.

4.1.3. Backend path

To edit the path to the workspace backend program(s) that was (were) installed, edit $GLOBUS_LOCATION/etc/workspace_service/jndi-config.xml

In jndi-config.xml, find the backendPath entry under the WorkspaceService section. Use an absolute path to the program. The service is distributed with the proper configuration for when the backend program is installed with the workspace-control default configurations and installer.

Note: if you are using the resource pool setup and the XenSSH implementation, this backendPath entry refers to the program's location on the VMM nodes.

4.1.4. Factory policies

Currently, the factory can be configured with three simple policies concerning workspace deployment. Each value is listed in the WorkspaceFactoryService section of $GLOBUS_LOCATION/etc/workspace_service/jndi-config.xml

The defaultRunningTimeMinutes value is used if the DeploymentTime element of the deployment request is not specified.

The maxRunningTimeMinutes value can not be exceed by the value of the DeploymentTime element of the deployment request.

The WSRFResouceMinutesPastRunningTime value configures the default termination time of the WSRF resource representing the workspace. When a workspace is shutdown, its representation is not terminated (one reason being that it can be started again). This value is added on to the running time and is currently only configurable at this location.

The optional architecture value can be one of the JSDL ProcessorArchitectureEnumeration values such as "x86". If you are running Xen on other architectures, this should be changed appropriately.

The optional vmm value will currently be Xen.

The optional vmmVersions value can be a comma separated list of one-word strings (deciding on values to use is a deployment issue within VOs).

The optional associationFileLocation value is a directory location that specifies where to find the networking pool information for the Allocate networking method. If this is a relative path, it is evaluated with respect to $GLOBUS_LOCATION. For more information on configuring associations, see below.

The optional certificateFileLocation value is a directory location that specifies where to find the certificate and keyfiles that an association may optionally refer to.

The optional resourcepoolFileLocation value is a directory location that specifies where to find the resource pool information for the xenSSH implementation. If this is a relative path, it is evaluated with respect to $GLOBUS_LOCATION. For more information on configuring a resource pool, see below.

For more information about the deployment request, see the TP1.2.2 interfaces documentation.

4.1.5. Optional: Resource pool configuration

This is optional because you may be using "local" mode where the service is being hosted on the only hypervisor. But in most cases, the resource pool mode of operation (where you deploy one grid service that manages many hypervisor nodes) is recommended. See Service Implementation.

As noted above, the resourcepoolFileLocation directory can contain files that represent resource pools. The file format is currently very simple: for each node in the pool, list the hostname and the amount of RAM it can spare for running guest VMs.

These configurations are currently under the same constraint as the association entries with respect to removing entries (see DB side note).

4.1.6. Optional: Networking assocations

The workspace service can allocate IP addresses from pools of addresses (called associations) for the Allocate networking method. The name of the file is the association name, and the directory to find all such files must be configured in the workspace factory JNDI config (see above).

The service is packaged with two sample association files public and private. These files can be found in the example associations directory: $GLOBUS_LOCATION/etc/workspace_service/associations. The files contain the syntax explanation as comments. Currently there is no script to auto-generate such files from higher level logic, each entry must be specified manually or generated via script.

Side note: The service takes these files and inserts them into its database (uniqueness is judged by the combination of association name and IP address). This happens during the container boot process in the factory service initialization. The files themselves are therefore not the authoritative source of association information, but are for administrator convenience. Note that currently there is no support for assessing if an administrator has removed any entries between boots of the GT container. If you have added any entries, it will add them, but there is no support for removing database entries besides manually deleting them via SQL or resetting the entire database (and losing all persistent state) by using "ant resetDB" from the service source tree. To implement such a feature would require a longer container boot process as well as shutting all workspace down that used the removed address (if there is user demand for this convenience, we can take the time to add it).

4.2. workspace-control program configuration

After untar-ing the package on the VMM node, move the sample configuration file to a permanent place and then edit it. There are many configurations for directory paths in the file and they almost all point to a hierarchy under /opt/workspace. These directories are for isolating and mounting the guest images as well as giving the program places for persistence, tmp files, and an image cache (where you can put the default kernel for example).

Every directory path is editable even when using the automatic installer because you will pass the configuration file to the installer. If this is your first time installing workspace-control it is probably best if you stick with the default /opt/workspace setup.

One of the directories is bin. The program is placed here by default, but importantly also tools that the program can call via sudo to make configuration changes before deployment. Because of this, the directory these tools are in and all of its parents must be owned by root. (Elevating an account's privileges to root in order to call a particular program over sudo when the program is editable by non-root defeats the purpose of using sudo.)

Note: the workspace-control installer can check all of the related permissions for you, create missing directories, and then adjust their permissions to be correct. There is also a detailed permissions note available in the workspace-control configuration file.

One of the necessary configurations to make is to point the program at the default kernel it should use for guests, this is one of the labelled options in the workspace-control configuration file. Another necessary configuration is to make sure its association (network pool) configurations match the workspace service's, this section is also labelled in the default configuration file.

You need to have a privileged user and unix group that will manage the VMs since we do not allow the whole program to be run as root. You can just pass the user and group information to the installer so it can know how to chown the directories.

For example, if you accept the default /opt/workspace hierarchy and your account and group are globus and globus, you could run these commands (roughly):

cd workspace-control
cp worksp.conf.example /opt/workspace/worksp.conf
./install.py -c /opt/workspace/worksp.conf -a globus -g globus -n
mount /boot
cp /boot/vmlinuz-2.6.16-xen /opt/workspace/images/
umount /boot

(now edit the /opt/workspace/worksp.conf file to point to that kernel for guest use; also, adjust the networking information)

Just before exiting, the installer will print out the sudo policies you need to configure (see the next section).

4.2.1. sudo

For Xen3 control, editing the networking configurations, and for one feature where the image itself is modified (certificate staging), the sudo program must be installed on the same node as the backend program. For more information about the mentioned networking configuration, see below.

As root with the visudo command, add these lines to the /etc/sudoers file that reflects the user running the GT container (the first word) and the correct full paths to the dhcp-config.sh tool, the xm tool, and the xend daemon (needed in the case where the workspace-control program is permitted to reboot the daemon if it has fallen).

Note: the installer will print out the exact sudo policy you need that matches the configurations it read in from the workspace-control conf file. e.g.:

You must manually add these policies to your sudo config (use visudo):

globus ALL=(root) NOPASSWD: /opt/workspace/bin/dhcp-config.sh
globus ALL=(root) NOPASSWD: /usr/sbin/xm
globus ALL=(root) NOPASSWD: /usr/sbin/xend

4.2.2. Authorized kernels

This version of the workspace service does not support unauthorized client kernels. As such, the guestkernels configuration under the [images] section lists the kernels workspaces are allowed to be instantiated with. A client can choose from these kernels in the metadata, but they must already exist at the hypervisor node and must be in the guestkernels list.

4.2.3. Networking

With Xen, for each NIC on the hypervisor node, it is typical to create a corresponding bridge such that guest virtual machines may be bridged to each NIC. This allows workspaces to be on different networks or allows spreading out of the networking load (in the case where the hypervisor's multiple NICs are on the same network).

Support for multiple physical and virtual NICs is present in this version, mapped in the grid context in terms of a networking association string. A client will specify the required association in the metadata, assuming the deployment nodes support that association (otherwise, a creation fault will be thrown). An association is created by configuring an entry for it in the workspace-control configuration file. This should match the service association configurations.

For example, the node could have two physical NICs, one bound to a private LAN and one bound to the Internet. To accomodate virtual networking cards being bound to both NICs, a site admin could create two bridges, xenbr0 and xenbr1. Then an association would be configured for each bridge, e.g., one called "public" and one called "private" for example (a typical configuration).

In the sample workspace-control configuration file, find the association_ settings in the [networking] section. If xenbr0 is the 'public' interface, add this line:

association_0: private; xenbr0; vif0.0 ; AA:01; 192.168.0.0/24

This specifies that there will be an association called 'private' with bridge 'xenbr0' with MAC address prefixes of 'AA:01' given to virtual NICs bound to this bridge and with an authorized IP range of 192.168.0.0 to 192.168.0.255.

The third field lists the interface of the NIC that is listening for DHCP requests. This is the interface that DHCP requests originating from workspaces should be allowed to broadcast to. Directing the request to a specific interface prevents DHCP requests from being broadcast to other workspaces as well as to the real network. You can doublecheck what vif is on what bridge by running "brctl show". Normally you'd run the DHCP server in dom0 and the defaults are like so:

xenbr0 - vif0.0
xenbr1 - vif0.1
xenbr2 - vif0.2

You are not required to run the DHCP server in dom0 as the workspaces' DHCP request broadcast can be directed to any interface, but currently to run it in another domain would require altering the dhcp-config.sh callout to work remotely (this would not be hard for you to do, grep for "dhcpconfigpath" in the xen_v2.py module and prepend ssh parameters, for example).

Important

The MAC address cannot have a truly random prefix.

Quoting the current entry at http://wiki.xensource.com/xenwiki/XenNetworking

When choosing MAC addresses to use, ensure you choose a unicast address.
That is, one with the low bit of the first octet set to zero. For example, an
address starting aa: is OK but ab: is not. It is best to keep to the range of
addresses declared to be "locally assigned" (rather than allocated globally to
hardware vendors). These have the second lowest bit set to one in the first
octet. For example, aa: is OK, a8: isn't.

In summary, an address of the following form should be OK:

XY:XX:XX:XX:XX:XX

where X is any hexadecimal digit, and Y is one of 2, 6, A or E

Note: if you only have one bridge for virtual network cards, a simple configuration would be to list one association like so:

association_0: default; AA; 1.0.0.0/16

Note that no bridge name is given, the default bridge as determined by Xen will be used. Note the IP range is fake; if you want to not keep track of workspace IP assignments (to prevent conflicts) or to not check if a requested IP setting is valid for the bridge, set the check_ip_ranges, track_MAC_assignments, and track_IP_assignments settings to 'false'.

4.2.4. In-VM Networking

To actually enact networking changes (using the Allocate networking method for example), the VM must set its own L3 networking information (IP address, default gateway, etc) from inside the VM. Currently we only support delivery of the information via DHCP. Booting into DHCP client mode is well supported in virtually every operating system in existence. Previously we passed information via kernel parameters which required a special understanding inside the VM. The result of using DHCP is that workspace images are easier to create and easier to maintain.

A DHCP server is required to run on each hypervisor node. The purpose of this server is to respond to broadcast requests from workspace's that are booting locally. Before starting a VM, if any of its NICs need to be configured via DHCP, workspace-control will call out to "dhcp-config.sh" via sudo, passing it a specific MAC address to IP mapping (as well as other information to be passed to the VM such as hostname, dns servers, broadcast address, default gateway, etc).

  • "Won't this interfere with my current DHCP server?" No.
  • "Will this cause unwanted packets on my physical LAN?" No.
  • "Will other workspaces be able to send broadcasts and get the wrong DHCP lease?" No.

In addition to a DHCP server, we also insert custom ebtables rules when the workspace is deployed. These rules accomplish three things:

  1. Only packets with the correct MAC address for this virtual interface are permitted.
  2. Broadcasted DHCP requests are only permitted to be bridged to the correct virtual interface (configured in workspace-control's configuration file, see the workspace-control networking configuration section.

  3. Only packets with the correct IP address for this virtual interface are permitted (as the NIC does not have an IP address yet when making a DHCP request, the IP check only happens if it is not a DHCP request.

A draft of the workspace DHCP design document is available here.

Configuring the DHCP server consists of copying the example DHCP file "dhcp.conf.example" to "/etc/dhcp/dhcpd.conf" and editing it to include the proper subnet lines (see the contents of the example file). The subnet lines are necessary to get the DHCP server to listen on the interface in the first place. No lease configurations, available ranges, etc. should be added: these are added dynamically to the file after the token at the bottom.

In most cases it is unecessary, but if you have a non-standard DHCP configuration you may need to look at the "dhcp-config.sh" script in the protected workspace bin directory and look at the "adjust as necessary" section. The assumptions made are as follows:

  • DHCP policy file to adjust: "/etc/dhcp/dhcpd.conf"
  • Stop DHCP server: "/etc/init.d/dhcpd stop"
  • Start DHCP server: "/etc/init.d/dhcpd start"
  • The standard unix utility "dirname" is assumed to be installed. This is used to find the workspace-control utilities "dhcp-conf-alter.py" and "ebtables-config.sh", we assume they are in the same directory as "dhcp-config.sh" itself. Paths to these can alternatively be hardcoded to fit your preferred configuration.

5. Testing

To test the workspace service, you can use the sample metadata and deployment files and a test image.

For a walkthrough and explanations, see the Workspace User Guide.

6. Troubleshooting

Any questions can be posted to the workspace-user mailing list and will likely be answered promptly by a member of the community. For instructions on how to subscribe and post messages to this list, see this web page.

  • Problem: Sometimes, from the start of the workspace's deployment, one of the VM's NICs is unreachable. (specifically, the ARP protocol does not resolve the IP address to a MAC address)

    Solution: Make sure the MAC address prefix is valid. See the backend networking configuration section for more details and the right setting to use.