Note

The Grid Community Toolkit documentation was taken from the Globus Toolkit 6.0 documentation. As a result, there may be inaccuracies and outdated information. Please report any problems to the Grid Community Forums as GitHub issues.

GCT → Admin Manuals → Installing GCT 6.2

Introduction

This guide is the starting point for everyone who wants to install Grid Community Toolkit 6.2. It will take you through a basic installation that installs the following basic services: a security infrastructure (GSI), GridFTP, and Execution Services (GRAM5).

Before you begin

Before you start installing the Grid Community Toolkit 6.2, there are a few things you should consider. The toolkit contains several components, and you may only be interested in some of them.

The Grid Community Toolkit version 6.2 includes:

  • GSI: security

  • GridFTP: file transfer

  • GRAM: job execution/resource management

  • MyProxy: credential repository/certificate authority

  • GSI-OpenSSH: GSI secure single sign-on remote shell

If you are new to the toolkit and want to experiment with the components, you may want to use a supported RedHat based or Debian based Linux system. With the new supported native packaging installs, they are the simplest platforms on which to install GCT services.

For the purposes of this documentation, GCT is being installed on a machine called elephant.

Typographical Conventions

Where there is a command to be typed, it will be preceded by one of the following prompts:

root# , root@donkey#

Run this command as the root super-user, on the elephant or donkey hosts respectively. You might have to use a command like su(8) or sudo(8) to start a root shell before executing the command.

myproxy%

Run this command as the myproxy user, on the elephant host. This user is created automatically when the myproxy-server package is installed.

quser% , quser@donkey%

Run this command as the normal user account you are intending to interact with your GCT sevices, on the elephant or donkey hosts. In this document, we use the quser accout for this, but if you have another user, you can use it for that purpose.

Commands themselves will be typeset as run-this-command -with-arguments, and responses to the commands like this Some Response Text. If there is some portion of a command which should be replaced by value, such as a version number, it will be typeset like this: REPLACEME.

Finally, in some cases you will be prompted for a passphrase. When that occurs, the entry of the passphrase will be indicated by ******, even though nothing will be printed to the screen.

Installing GCT 6.2

Installing Binary Packages

Prerequisites

The Grid Community Toolkit 6.2 is available as a set of RPM packages provided by EPEL and Fedora package repositories and Debian packages provided by Debian and Ubuntu package repositories for Linux distributions as well as a source installer for the whole GCT and separate source packages for each GCT component provided by the GridCF from our repository which can be used both on Linux and on other operating systems. Please see the bottom of the GCT release page for information about the supported Linux distributions. In this quickstart, we will be installing RPM packages. Thus, it is a prerequisite for following this quickstart that you are running a distribution for which RPMs are available. If you are running a supported Debian or Ubuntu system, the process is very similar, but you’ll need to use the apt[-get] command or similar tools to install the packages. For the source installer, there is more work involved, and you’ll need to consult the full installation guide.

First, we will set up our system to use the EPEL package repository. This repository contains the GCT software packages for Linux distributions like Red Hat Enterprise Linux (RHEL) 7, 8 and 9, Community Enterprise Operating System (CentOS) 7 and CentOS Stream 8 and 9, RockyLinux 8 or Scientific Linux 7. If your distribution has GCT 6.2 packages within its default repositories - like Fedora has - you can skip to the next section.

The respective EPEL repo RPMs for the respective OS versions are linked from the EPEL documentation website.

To install binary RPMs, you first need to install the EPEL repo RPM matching your OS. E.g. like so:

root# yum install epel-release

…​for CentOS 7 which has it included in its default repositories.

For other OS versions follow the EPEL quickstart documentation.

For Fedora and also Debian and Ubuntu the GCT packages are included in the default package repositories, so no further action is required there to be able to install GCT packages.

You can now use your operating system’s packaging tools: yum (or dnf) or apt[-get], to install the GCT components.

Installing the Toolkit on Linux

The components of the toolkit can be installed separately, or all at once. This section will show how to install various components, on both RPM based and Debian based Linux systems.

For Fedora or Red Hat-based systems, use the yum or dnf commands to install the GCT components and their dependencies. For SUSE Linux distributions, use zypper. For Debian-based systems, used the apt[-get] or aptitude commands.

For example, to install the GridFTP client tools, do the following for Red Hat-based systems:

root# yum install \
    globus-common-progs globus-gsi-cert-utils-progs globus-ftp-client globus-gass-copy-progs globus-proxy-utils

Do the following for Debian-based systems:

root# apt-get install \
    globus-common-progs globus-gsi-cert-utils-progs globus-ftp-client globus-gass-copy-progs globus-proxy-utils
Updating a GCT Installation

For the GCT the package repositories in general always include the GCT component versions that were included in the latest available GCT version, except for Debian, where the package repositories for the stable distributions include the GCT component versions that were included in the latest GCT version that was available before the freeze for the respective stable version of Debian took place. Debian unstable (aka Sid) instead always includes the GCT component versions that were included in the latest available GCT version like the other Linux distributions.

In addition fixes for broken GCT packages or security fixes for GCT components are included in the package repositories also in between GCT releases.

In general updating a GCT installation can hence be done via yum (or dnf) or apt[-get] manually or happens automatically during unattended OS upgrades.

Installation from Source Installer

Note

Installing using the Source Installer is only recommended on platforms for which native packages are not available. If you are installing onto a Red Hat or Debian based Linux system, please see the section above.

Software Prerequisites

Required software

To build the Grid Community Toolkit from the source installer, first download the source from the GCT Releases page on GitHub, and be sure you have all of the following prerequisites installed.

This table shows specific package names (where available) for systems supported by GCT 6.2:

Prerequisite Reason RedHat-based Systems Debian-based Systems Solaris 11 Mac OS X

C Compiler

Most of the toolkit is written in C, using C99 and POSIX.1 features and libraries.

gcc

gcc

pkg:/developer/gcc-45 or Solaris Studio 12.3

XCode

GNU or BSD sed

Standard sed does not support long enough lines to process autoconf-generated scripts and Makefiles

sed

sed

pkg:/text/gnu-sed

(included in OS)

GNU Make

Standard make does not support long enough lines to process autoconf-generated makefiles

make

make

pkg:/developer/build/gnu-make

(included in XCode)

OpenSSL 0.9.8 or higher

GSI security uses OpenSSL’s implementation of the SSL protocol and X.509 certificates.

openssl-devel

libssl-dev

pkg:/library/security/openssl

(included in base OS)

Perl 5.10 or higher

Parts of GRAM5 are written in Perl, as are many test scripts

perl

perl

pkg:/runtime/perl-512

(included in base OS)

pkg-config

Parts of GRAM5 are written in Perl

pkgconfig

pkg-config

pkg:/developer/gnome/gettext

Download and install from freedesktop.org source packages
Note

In order to use the GNU versions of sed, tar, and make on Solaris, put /usr/gnu/bin at the head of your path. Also, to use all of the perl executables, add at the head of your path. Also, to use all of the perl executables, add /usr/perl5/bin to your path. to your path.

Installing from Source Installer

  1. Create a user named globus. This non-privileged user will be used to perform administrative tasks, deploying services, etc. Pick an installation directory, and make sure this account has read and write permissions in the installation directory.

    Tip

    You might need to create the target directory as root, then chown it to the globus user:

    root# mkdir <INSTALL_DIR>
root# chown globus:globus <INSTALL_DIR>
    Important

    If for some reason you do not create a user named globus, be sure to run the installation as a non-root user. In that case, make sure to pick an install directory that your user account has write access to.

  2. Download the required software noted in Software Prerequisites.

  3. The Grid Community Toolkit Source Installer sets the installation directory by default to /usr/local/globus-6, but you may replace /usr/local/globus-6 with whatever directory you wish to install to, by setting the prefix when you configure.

    As the globus user, run:

    globus% ./configure --prefix=<INSTALL_DIR>

    You can use command line arguments to ./configure for a more custom install.

    For a full list of options, see ./configure --help.

  4. The source installer will build all of the Grid Community Toolkit packages in the default make rule. The following Makefile targets can be used to build subsets of the Grid Community Toolkit:

    ccommonlibs

    C Common Libraries

    gridftp

    GridFTP Client and Server

    gsi

    Security Libraries and Tools

    gsi

    Security Libraries and Tools

    udt

    Globus XIO UDT Driver

    myproxy

    MyProxy Client and Server

    gsi-openssh

    GSI OpenSSH Client and Server

    gram5

    GRAM5 Client and Libraries

    gram5-server

    GRAM5 Service

    gram5-lsf

    GRAM5 LSF Adapter

    gram5-sge

    GRAM5 SGE Adapter

    gram5-slurm

    GRAM5 SLURM Adapter

    gram5-condor

    GRAM5 Condor Adapter

    gram5-pbs

    GRAM5 PBS Adapter

    gram5-auditing

    GRAM5 Auditing Support

    Run:

    globus% make

    Note that this command can take a while to complete. If you wish to have a log file of the build, use tee:

    globus% make 2>&1 | tee build.log

    The syntax above assumes a Bourne shell. If you are using another shell, redirect stderr to stdout and then pipe it to tee.

  5. To test the toolkit, or particular packages within the toolkit, run:

    globus% make check

    or

    globus% make <COMPONENT>-check

    where <COMPONENT> is the name of the package to test. As an example, you could run

    globus% make globus_gssapi_gsi-check

    to run the GSSAPI test programs.

  6. Finally, run:

    globus% make install

    This completes your installation. Now you may move on to the configuration sections of the following chapters.

    We recommend that you keep your installation up to date and always use the latest version of the GCT. You may also be interested in subscribing to the mailing lists listed under Contact and News on the GridCF website.

Updating an Installation

The updates available in the native packages described above are also published as source packages in our repository. To install update packages, follow their download link, untar them, then configure them with the same prefix as your original installation and install them as described above.

Basic Security Configuration

Obtain host credentials

You must have X.509 certificates to use the GCT 6.2 software securely (referred to in this documentation as host certificates). For an overview of certificates for GSI (security) see GSI Configuration Information and GSI Environment Variables.

If you will need to be interoperable with other sites, you will need to obtain certs from a trusted Certificate Authority, such as those that are included in IGTF. If you are simply testing the software on your own resources, SimpleCA offers an easy way to create your own certificates (see section below).

Host credentials must:

  • consist of the following two files: hostcert.pem and and hostkey.pem

  • be in the appropriate directory for secure services: /etc/grid-security/

  • match the hostname for a the machine. If the machine is going to be accessed remotely, the name on the certificate must match the network-visible hostname.

You have the following options:

Request a certificate from an existing CA

Your best option is to use an already existing CA. You may have access to one from the company you work for or an organization you are affiliated with. Some universities provide certificates for their members and affiliates. Contact your support organization for details about how to acquire a certificate. You may find your CA listed in the TERENA Repository.

If you already have a CA, you will need to follow their configuration directions. If they include a CA setup package, follow the CAs instruction on how to install the setup package. If they do not, you will need to create an /etc/grid-security/certificates directory and include the CA cert and signing policy in that directory. See directory and include the CA cert and signing policy in that directory. See Configuring a Trusted CA for more details.

This type of certificate is best for service deployment and Grid inter-operation.

SimpleCA

SimpleCA provides a wrapper around the OpenSSL CA functionality and is sufficient for simple Grid services. Alternatively, you can use OpenSSL’s CA.sh command on its own. Instructions on how to use the SimpleCA can be found in Installing SimpleCA.

SimpleCA is suitable for testing or when a certificate authority is not available.

If you install the globus-simpleca native package, it will automatically create a CA and host certificate if you don’t have one configured yet. Otherwise, you’ll need to use grid-ca-create to create the CA and grid-default-ca to make that the default for requesting credentials.

To create user credentials, you can run the command grid-cert-request as a user that you want to create a credential for. You can then run the grid-ca-sign command as the simpleca user to sign the certificate.

Add authorization

Installing GCT services on your resources doesn’t automatically authorize users to use these services. Each user must have their own user certificate, and each user certificate must be mapped to a local account.

To add authorizations for users, you’ll need to update the grid-mapfile database to include the mapping between the credentials and the local user accounts. database to include the mapping between the credentials and the local user accounts.

You’ll need two pieces of information:

  • the subject name of a user’s certificate

  • the local account name that the certificate holder can access.

To start with, if you have created a user certificate, you can run the grid-cert-info command to get the certificate’s subject name, and id -un to get the account name:

globus% grid-cert-info -subject
/O=Grid/OU=GlobusTest/OU=simpleCA-elephant.globus.org/CN=Globus User
globus% id -un
globus

You may add the line by running the following command as root:

root# grid-mapfile-add-entry \
    -dn "/O=Grid/OU=GlobusTest/OU=simpleCA-elephant.globus.org/CN=Globus User" \
    -ln gtuser
Modifying /etc/grid-security/grid-mapfile ...
/etc/grid-security/grid-mapfile does not exist... Attempting to create /etc/grid-security/grid-mapfile
New entry:
"/O=Grid/OU=GlobusTest/OU=simpleCA-elephant.globus.org/CN=Globus User" globus
(1) entry added
Important

The quotes around the subject name are important, because it contains spaces.

Verify Basic Security

Now that you have installed a trusted CA, acquired a hostcert and acquired a usercert, you may verify that your security setup is complete. As your user account, run the following command:

gtuser$ grid-proxy-init -verify -debug

User Cert File: /home/gtuser/.globus/usercert.pem
User Key File: /home/gtuser/.globus/userkey.pem

Trusted CA Cert Dir: /etc/grid-security/certificates

Output File: /tmp/x509up_u506
Your identity: /DC=org/DC=doegrids/OU=People/CN=GT User 332900
Enter GRID pass phrase for this identity:
Creating proxy ...++++++++++++
..................++++++++++++
 Done
Proxy Verify OK
Your proxy is valid until: Fri Jan 28 23:13:22 2005

There are a few things you can notice from this command. Your usercert and key are located in $HOME/.globus/. The proxy certificate is created in . The proxy certificate is created in /tmp/. The "up" stands for "user proxy", and the . The "up" stands for "user proxy", and the _u506 will be your UNIX userid. It also prints out your distinguished name (DN), and the proxy is valid for 12 hours.

If this command succeeds, your single node is correctly configured.

If you get an error, or if you want to see more diagnostic information about your certificates, run the following:

gtuser$ grid-cert-diagnostics

For more troubleshooting information, see the GSI troubleshooting guide

Firewall configuration

There are four possible firewall scenarios that might present themselves: restrictions on incoming and outgoing ports for both client and server scenarios.

This section divides sites into two categories: client sites, which have users that are acting as clients to Grid services, and server sites, which are running Grid services. Server sites also often act as client sites either because they also have users on site or jobs submitted by users to the site act as clients to other sites by retrieving data from other sites or spawning sub-jobs.

Client Site Firewall Requirements

This section describes the requirements placed on firewalls at sites containing Grid Community Toolkit clients. Note that often jobs submitted to sites running GCT services will act as clients (e.g. retrieving files needed by the job, spawning subjobs), so server sites will also have client site requirements.

Allowed Outgoing Ports

Clients need to be able to make outgoing connections freely from ephemeral ports on hosts at the client site to all ports at server sites.

Allowed Incoming Ports

As described in Job State Callbacks and Polling, the Grid Community Toolkit GRAM service uses callbacks to communicate state changes to clients and, optionally, to stage files to/from the client. If connections are not allowed back to the Grid Community Toolkit clients, the following restrictions will be in effect:

  • You cannot do a job submission request and redirect the output back to the client. This means the globus-job-run command won’t work. globus-job-submit will work, but you cannot use globus-job-get-output. globusrun with the -o option also will not work.

  • Staging to or from the client will also not work, which precludes the -s and -w options.

  • The client cannot be notified of state changes in the job, e.g. completion.

To allow these callbacks, client sites should allow incoming connection in the ephemeral port range. Client sites wishing to restrict incoming connections in the ephemeral port range should select a port range for their site. The size of this range should be approximately 10 ports per expected simultaneous user on a given host, though this may vary depending on the actual usage characteristics. Hosts on which clients run should have the GLOBUS_TCP_PORT_RANGE environment variable set for the users to reflect the site’s chosen range.

Network Address Translation (NAT)

Clients behind NATs will be restricted as described in Allowed Incoming Ports unless the firewall and site hosts are configured to allow incoming connections.

This configuration involves:

  • Select a separate portion of the ephemeral port range for each host at the site on which clients will be running (e.g. 45000-45099 for host A, 45100-45199 for host B, etc.).

  • Configure the NAT to direct incoming connections in the port range for each host back to the appropriate host (e.g., configure 45000-45099 on the NAT to forward to 45000-45099 on host A).

  • Configure the Grid Community Toolkit clients on each site host to use the selected port range for the host using the techniques described in If client is behind a firewall.

  • Configure Grid Community Toolkit clients to advertise the firewall as the hostname to use for callbacks from the server host. This is done using the GLOBUS_HOSTNAME environment variable. The client must also have the GLOBUS_HOSTNAME environment variable set to the hostname of the external side of the NAT firewall. This will cause the client software to advertise the firewall’s hostname as the hostname to be used for callbacks causing connections from the server intended for it to go to the firewall (which redirects them to the client).

Server Site Firewall Requirements

This section describes firewall policy requirements at sites that host Grid services. Sites that host Grid services often host Grid clients, however the policy requirements described in this section are adequate for clients as well.

Allowed Incoming Ports

A server site should allow incoming connections to the well-known Grid Service Ports as well as ephemeral ports. These ports are 22/tcp (for gsi-enabled openssh), 2119/tcp (for GRAM) and 2811/tcp for GridFTP.

A server not allowing incoming connections in the ephemeral port range will have the following restrictions:

  • If port 2119/tcp is open, GRAM will allow jobs to be submitted, but further management of the jobs will not be possible.

  • While it will be possible to make GridFTP control connections if port 2811/tcp is open, it will not possible to actually get or put files.

Server sites wishing to restrict incoming connections in the ephemeral port range should select a range of port numbers. The size of this range should be approximately 20 ports per expected simultaneous user on a given host, though this may vary depending on the actual usage characteristics. While it will take some operational experience to determine just how big this range needs to be, it is suggested that any major server site open a port range of at least a few hundred ports. Grid Services should configured as described in Section to reflect the site’s chosen range.

Allowed Outgoing Ports

Server sites should allow outgoing connections freely from ephemeral ports at the server site to ephemeral ports at client sites as well as to Grid Service Ports at other sites.

Network Address Translation (NAT)

Grid services are not supported to work behind NAT firewalls because the security mechanisms employed by GCT require knowledge of the actual IP address of the host that is being connected to.

We do note there have been some successes in running GCT services behind NAT firewalls.

Summary of Grid Community Toolkit Traffic

Table 1. Summary of Grid Community Toolkit Traffic
Application Network Ports Comments

GRAM Gatekeeper(to start jobs)

To 2119/tcp on server from controllable ephemeral port on client

Connections back to client (controllable ephemeral port to controllable ephemeral port) required if executable or data staged from client or output from job sent back to client. Port 2119/tcp defined by IANA

GRAM Job-Manager

From controllable ephemeral port on client to controllable ephemeral port on server.

Port on server selected when original connection made by the client to the Gatekeeper and returned to the client in a URL. May result in connection back to client from ephemeral port on server to controllable ephemeral port on client.

GridFTP

From controllable ephemeral port on client to port 2811/tcp on server for control channel.

Port 2811/tcp defined by IANA.

GSI-Enabled SSH

From ephemeral port on client to port 22/tcp on server.

Same as standard SSH. Port 22/tcp defined by IANA.

MyProxy

From ephemeral port on client to port 7512/tcp on server.

Default. Can be modified by site.

Controlling The Ephemeral Port Range

Controllable ephemeral ports in the Grid Community Toolkit can be restricted to a given range. setting the environment variable GLOBUS_TCP_PORT_RANGE can restrict ephemeral ports. The value of this variable should be formatted as min,max (a comma separated pair). This will cause the GCT libraries (specifically GlobusIO) to select port numbers for controllable ports in that specified range.

% GLOBUS_TCP_PORT_RANGE=40000,40010
% export GLOBUS_TCP_PORT_RANGE
% globus-gass-server
https://globicus.lbl.gov:40000
^C
%

This environment variable is respected by both clients and servers that are started from within the environment in which it is set. There are better ways, however, to configure a globus-job-manager or a GridFTP server to restrict its port range.

  • globus-job-manager has an option, -globus-tcp-port-range PORT_RANGE that acts in the same manner as the environment variable. It can be specified on the command line or in the configuration file. See the job manager documentation for all of its options.

  • See the GridFTP documentation for information about using GridFTP with firewalls.

Basic Setup for GCT 6.2

The Quickstart Guide walks you through setting up basic services on multiple machines.

Appendix

The Install Guide appendix can be found here.