Globus Toolkit JIRA Archive

To reproduce, provide guc with a singe line input file, use the sync option, and dump-only option. (non-recursive)

Specify a source and dest path that is a directory but do not put a trailing / in.

Guc will output as if there is a file that needs to be synced.

Code in globus_l_guc_expand_single_url() checks for a trailing slash to determine if target is a directory.  The stat type field is not inspected.

I'm going to put in a patch for globus online so that an error is returned.

A regression in 5.2.1 causes config values set to relative paths to be invalid.
example: these should result in the same behavior, but 1) will fail.

1) cd /tmp; globus-gridftp-server -l logfile

2) cd /home; globus-gridftp-server -l /tmp/logfile

This is my script.  It's failing in /Users/karl/koa/lta/mac_gui/ext-globus/gt5.0.2-all-source-installer/source-trees/xio/src (amont other places) because configure ignores what I set for CC.

MACOSX_SDK=/Developer/SDKs/MacOSX10.4u.sdk
export CC=/usr/bin/gcc-4.0
export GLOBUS_CC=$CC
export CFLAGS="-DMACOSX_DEPLOYMENT_TARGET=10.4 -isysroot $MACOSX_SDK \
        -mmacosx-version-min=10.4"
export LDFLAGS="-isysroot $MACOSX_SDK"
export LIBTOOL="/tmp/gtscratch/sbin/libtool-gcc32dbg --tag=CC"

#rm -rf /tmp/gtscratch
./configure --prefix=/tmp/gtscratch \
    --disable-system-openssl \
    --with-flavor=gcc32dbg \
    --with-gsiopensshargs='--without-pam'

make gpt
make globus_core
make globus-data-management-server

Please make gridftp work with an alternate CC.  This is needed so I can set Mac OS 10.4+ as the ABI.

guc has different queues for urls from file input and urls from listing.  when only the input queue has urls, pipelining is very inefficient and only a small percentage of urls get pipelined.

this is somewhat related to #KOA-1085 in that only the few pipelined urls exhibit the bug in that issue.

Dear all,

during installation of GridFTP (from GT 5.0.4) on an OpenBSD 4.9 x86_64 virtual machine, I stumbled upon a problem that blocks successful compilation.

I extracted the Globus Toolkit 5.0.4 sources, configured them with:

$ ./configure --prefix=$GLOBUS_LOCATION --with-flavor=gcc32

...and tried to compile and install GridFTP with:

$ time make gridftp

The make run starts with building the "gpt" target. But this fails after some time with the following message:

"
cd gpt && OBJECT_MODE=32 ./build_gpt
[...]
build_gpt ====> building /home/globus/tmp/gt5.0.4-all-source-installer/gpt/packaging_tools
/home/globus/usr/local/globus/sbin/gpt-build -srcdir=source-trees-thr/core/source gcc64dbgthr
sh: NOT: not found
/home/globus/usr/local/globus/etc/gpt/globus_core-src.tar.gz could not be untarred:512
Died at /home/globus/usr/local/globus/lib/perl/Grid/PkgMngmt/ExpandSource.pm line 42.
make: *** [globus_core-thr-compile] Error 2
"

Checking "[...]/ExpandSource.pm" it calls an external function "get_tool_location()" from a non-existing Perl module named "LocalEnv.pm". There only exists a file "LocalEnv.pm.in" both in the sources dir ($GLOBUS_SOURCE/gpt/packaging_tools/perl/GPT") and in the installation dir ($GLOBUS_LOCATION/lib/perl/Grid/GPT").

I don't know if Perl tries to use this module, or just refuses it, because it cannot find the *.pm file for it. The *.pm.in file still contains placeholders like "@@" for gtar, gunzip and others, that weren't replaced by the actual paths to the programs. I assume that this *.pm.in file isn't there by intention. BTW, there are other *.pm.in files located in the same "[...]/packaging_tools/perl/GPT" dir, but the corresponding *.pm files exist.

To workaround this issue, I just replaced the calls to "get_tool_location()"  for "$gunzip = [...] " and "$gtar = [...]" in lines 83/84 of "[...]/ExpandSource.pm" with the actual paths to the tools. Then making the gpt target could continue successfully. The remainder of the compilation also works through. The gridftp target is built successfully.

I don't have much insight to the GPT tools, but I assume there is an error in the process that prepares the Perl modules. Could you have a look?

Best regards,
Frank Scheiner

--
Frank Scheiner

High Performance Computing Center Stuttgart (HLRS)
Department Project User Management & Accounting

Dear all,

I've developed a usage numbers collection toolkit using the netlogger style logs provided by the Globus GridFTP service.
Information about GridFTP operations is stored in an SQL DB. This database can be queried later to draw some useful information
from the logged data. For example:

* aggregated traffic for the last month (or another period)
* average performance of transfers
* top senders
* etc.

During testing I recognized some issues with the logfiles that hinder the collection or make it even impossible:

Tests with striped transfers showed that a striped transfer is only logged on one "stripe" (usually the last stripe) and the frontend's netlog. The problem is, that this "stripe" only logs the amount of data transferred by itself and not the whole data that was transferred by all stripes. If one wants to know the full size, one has to gather this information from the netlog on the frontend. But the netlog on the frontend doesn't log the user name (if it's not running as root - please correct me if I'm wrong). Additionally the START and DATE values (on frontend and backend) differ slightly, so one also cannot correlate the line from the frontend's netlog with the corresponding line of the last "stripe" to get the user name.
In PRACE we like to monitor availability and performance of our GridFTP servers regularly, but it would be nice if we could filter this "monitoring" traffic. With the user name it would be easy, but as I described this is not really possible, as one can either have the user name (backend netlog) or the full amount of data that was transferred (frontend netlog).

It would be very nice, if either all the backends would log the amount of data they transferred, or if the frontend would log the username and the remote system's IP address (which is also missing in the frontend's netlog, as I recently found out, it's always "0.0.0.0").

Is the current behaviour intented? If you need more details, please let me know.

Best regards,
Frank Scheiner

--
Frank Scheiner

High Performance Computing Center Stuttgart (HLRS)
Department Project User Management & Accounting

1) globus-url-copy -rst -rst-retries 15 -rst-interval 1  -st 10 ... 
2) stop the network after a while (e.g. by command: service network stop)
3) the application crashes just after the retries:

Segmentation Fault

The bug is reproducable in 100%

Comment ad 1: the crash occurs if the time needed for retries > stall timeout
In the above use case:
15 retries * (1 sec of interval + time for single retry) > 10 sec of stall timeout
The problem may be workarounded by estimating the parameters, but the we never know "time for single retry" exactly,
so that potentially any coexistence of -rst and -st is dangerous.

Comment ad 2: probably the same problem will occur in case of other communication errors, but this one is easy to simulate

GT was installed from rpm epel repository, (rpm version for Centos: globus-ftp-client.x86_64 0:5.3-2.el5),
but I expect the same problem with source-compilled version.

In globus_ftp_client_operationattr_set_authorization(), if a strdup() fails, i_attr->auth_info.account ends up pointing to freed memory. Specifically, if the strdup() of subject fails, the restoring of i_attr->auth_info.account isn't done correctly. The problem arises if i_attr->auth_info.account isn't NULL, account is NULL, and subject isn't NULL.
I'll attach a patch.

If I'm not the owner of a directory, but a member of the group who owns the directory, the server does not show me permissions of the directory via MLST.

Example:
The directory chi-vm-4.isi.edu:/test/phaseI is owned by birn-dwei:fbirn_it_docs. I'm not birn-dwei, but I'm in the group fbirn_it_docs, and the permissions of that directory are 0770.

Snipped from the server-logfile:
[27614] Tue Nov 10 07:26:20 2009 :: 152.16.51.164:52017: [CLIENT]: MLST /test/phaseI
[27614] Tue Nov 10 07:26:20 2009 :: 152.16.51.164:52017: [SERVER]: 250-status of /test/phaseI
 Type=dir;Modify=20091109221317;Size=4096;Perm=;UNIX.mode=0770;UNIX.owner=birn-dwei;UNIX.group=fbirn_it_docs;Unique=810-b58001; /test/phaseI

RFC 3659 says in section 7.5.5:
   7.5.5. The perm Fact
       The perm fact is used to indicate access rights the current FTP user
       has over the object listed. ...

Perm probably shouldn't be empty. If I own a directory, perm contains the right permissions.

1) stop the network (e.g. service network stop)
2) run globus-url-copy with any "-rst" option (e.g. globus-url-copy -rst -rst-retries 15 -rst-interval 1 ...)
3) the application crashes just as the retries:

Segmentation Fault

The bug is reproducable in 100%

GT was installed from rpm epel repository, (rpm version for Centos: globus-ftp-client.x86_64 0:5.3-2.el5),
but I expect the same problem with source-compilled version.

GFDL has a wrapper on top of guc that does retry transfers when guc exits with an error. As GUC is exiting with zero when it hits a stall-timeout, the tool wouldn't catch the error and retry the transfer.

Scott Koranda has created a Python GridFTP client that is a thin wrapper around our C client.  We would like to enhance that using GRIDFTP-60 (e.g. "C client v42, Python wrapper v5"), and then ship it as part of GT.  This would allow us to monitor usage of this Python client over time.  If we are seeing significant adoption, we can take over support for it ourselves and make it a standard part of Globus.

feature request or a bug :-)

I see someone else cares enough to write a patch to preserve timestamps though the patch does not separate out this issue. See:
http://gridftp.bio-mirror.net/biomirror/
http://gridftp.bio-mirror.net/biomirror/gt5.0.2_patches.txt

Condor has been using the gridftp client library to interact with the NorduGrid ARC job scheduler. ARC uses the gridftp procotol as its job submission/management interface. Submitting a new job includes putting a file containing its JDL (job description language). The status of submitted jobs can be monitored by getting a series of files from the server.

In order to support submission to ARC, we had to expose an explicit CWD command in the gridftp client API. In ARC, to submit a new job, the client issues a CWD command to directory /jobs/new. The server's response prints a current working directory of 'jobs/########', where '########' is the id of the new job. The client then puts a file containing the job description to '/jobs/new/job'.

I would like to contribute our patch for inclusion in the the gridftp codebase. This would allow us to use the standard gridftp libraries instead of maintaining our own patched version. The patch may also be of value to other users. The patch was made against Globus 5.0.0. Let me know if you have any questions or concerns.

GFDL has requested this feature.

GFDL is using a authz callout for LDAP (developed by suragrid) in GridFTP. There is no support for this callout. They want such a callout be included in GT.

associate each url with a credential set, possibly via existing alias mechanism

We have some colleagues who would benefit from using UDT (they have a high-bandwidth link, but a relatively high rate of packet loss).  Currently, OSG doesn't ship UDT because RPM support is missing in Globus.

I started an email thread about it a few months back, but I don't think it ever resulted in a ticket.  One sticking point was an RPM of UDT itself.  I have a spec file, attached, that packages UDT.

Once that was done, I got lost in the depths of Globus packaging and wasn't able to get a UDT plugin for GridFTP built.

Document specification of commands/protocol that are added in GT GridFTP implementation. i.e, the commands that are not available in OGF GridFTP v1 and v2 spec and the relevant RFCs

The GridFTP init script does not follow the fedora init conventions enough to be acceptable to EPEL, or to the OSG packaging effort. GRAM-241 has some info about things that were needed for the GRAM scripts---similar things will need to be done for the GridFTP init scripts. See http://fedoraproject.org/wiki/Packaging:SysVInitScript for more info on the fedora guidelines.

Bottleneck detection is not supported by most GridFTP servers. Main reason for this is this code is not build by default. We need to include this in default build as this is critical to identify the bottleneck for a transfer

Karl is implementing rm in GO and he runs into this issue. As MLSD does not return broken symlinks, he has to use SITE RDEL and that has a bug too.

Without this information, the logs are less useful.

Many scientific communities have directories that consists of 10,000+ files. In the Globus Online web GUI, they would like to filter files in a directory using wild cards. We need support in GridFTP to list files using wild cards.

Determine the list of additional things that needs to be added to the usage statistics packet for GridFTP in the next release. Some of these include:
xio stack in use
gridftp session packet (session can consist of number of individual transfers)

The fix for GRIDFTP-185 to fix IPv6 compatibility required removing the data IP address from the EPSV response.  Apparently an earlier version of the spec made the address optional, while the final version prohibits it.  The spec intention is to make NAT traversal easier, but this breaks cases where we legitimately want to have different control channel and data channel addresses.

Some clients fail immediately when encountering the address in the EPSV response (google chrome), while the globus ftp client library silently ignores it and follows the intention of the spec, always connecting to the given port on the control channel address.

We previously extended our own SPAS command to address this issue, but the client library does not support this yet.  "SPAS [1|2]" will respond with (possible multiple) EPSV formatted data contact strings including an ipv4 (SPAS 1) or ipv6 (SPAS 2) ip address.

At some point we will need a solution in order to support both ipv6 and different control/data interfaces at the same time.

Adrian Colesa from the IGE project wrote some missing man pages for the globus-gridftp-server package: https://rt.ige-project.eu/rt/Ticket/Display.html?id=32
I attach these here.

From the "How OSG uses Globus" doc, it was suggested to evaluate if GridFTP performance could be improved when being used over emerging dedicated circuits (as opposed to shared best effort networks)

The Globus Online team was attempting recursive transfers for ESG using guc's dump -only sync option.  The user cert used did not have a CA trusted by GO.  Thus, guc would fail the handshake on the data channel and initiate the TCP close but the error given was:


Details       : error: Unable to list url gsiftp://cmip2.dkrz.de:2812/gpfs_750/transfer/replication_cmip5/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/piControl/6hr/atmos/6hrPlev/r1i1p1/v20101129/psl/:
globus_ftp_client: the server responded with an error
500 500-Command failed. : an end-of-file was reached
500-globus_xio: The GSI XIO driver failed to establish a secure connection. The failure occured during a handshake read.
500-globus_xio: An end of file occurred
500 End.


Command was:

running: ['/usr/local/globus/bin/globus-url-copy', '-src-cred', '/tmp/koauser1060.1001/tmptek_8rkoaproxy', '-dst-cred', '/tmp/koauser1060.1001/tmpjRR40akoaproxy', '-r', '-sync', '-sync-level', '2', '-do', '-', u'gsiftp://cmip2.dkrz.de:2812/gpfs_750/transfer/replication_cmip5/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES/piControl/6hr/atmos/6hrPlev/r1i1p1/v20101129/psl/', u'gsiftp://cmip-bdm1.badc.rl.ac.uk:2811/disks/drizzle1/archive/test-data/psl/']

This request came from Jason Alt at NCSA, and in the context of better integration of GO Transfer with their MSS. He stated that he had to modify the GridFTP server code to add custom commands, which is not idea, and would like to see a pluggable architecture there.

Rachana

This will enable older middleware to work with ipv6.

Should also enable globus_io support, or possibly change the default to allowed.

This has happened a couple times in globus online file transfer and it is really frustrating to figure out.   We try to allow any *user* cert and only do CA cert checks on hostnames (require igtf cas, etc.).  That works fine to log in to a gridftp server and do transfers.  However, when doing a directory listing that uses DCAU, the control channel lib (this happens with our new dirlist tool and guc) ends up looking at signing policy files for the user cert and bombs out if there's a problem .   It would be nice if the user cert is just marked as trusted, period, and dcau doesn't fail if a proxy issued by that cert is returned by the server.

On the GO side, we have had two cases thus far where directory listings failed and caused an obscure error (

globus_xio_gsi: gss_init_sec_context failed.
globus_gsi_gssapi: Unable to verify remote side's credentials
globus_gsi_gssapi: SSLv3 handshake problems: Couldn't do ssl handshake
OpenSSL Error: s3_pkt.c:1087: in library: SSL routines, function SSL3_READ_BYTES
: sslv3 alert unsupported certificate SSL alert number 43

One was where we didn't have a signing policy for a cred, the other was when the signing policy check didn't succeed.  (Actually, a third case where we didn't have the CA for the user cred).  All of those gave the fairly useless SSL3_READ_BYTES error from the *server*.

We would much rather have the *client* say "client error: signing policy check failed for cert X" or "signing policy X does not exist" to make this faster to debug.

From user complaints and from internal developer debugging, a number of error messages need to be improved

Enhance GridFTP framework to support and utilize network reservations, including a pluggable interface for existing
and future end-to-end network reservation services, integration of OSCARS and TeraPaths via GridFTP
plugins, support for advanced transport protocols via XIO modules.

At the CEDPS review in May '09, we have been asked to quantify the benefits of various GridFTP features, developed as part of the CEDPS project, to the user community. Analysis of usage statistics need to be improved to get this type of information in an automated fashion.

Firewalls pose a problem for data channel establishment in two-channel FTP-based protocols such as GridFTP. Common firewall configurations allow outbound connection requests but block all incoming connection requests. In other words, firewalls often block the path to a listener, thus making it impossible for the listening side of the FTP data channel to be properly contacted. Solutions such as opening a range of ports have been proposed but not embraced by security-conscious system administrators.
Here the idea of single port GridFTP is proposed. Server will listen on single port (2811) for both control and data channels.
- 2811 listener is a little more than current inetd-type process -- it will read one command from a connection to know whether to start a control or data process.
- new connections come in on 2811 and give control channel auth command, and control channel process is forked.
- on control channel, pasv-type command is sent, and response includes a host:2811:token.
- data connection comes in again on 2811, gives data command, and data process is forked.
- data chan process decrypts token with host cert or dummy stripe group cert, which tells it where to contact the control process.
- data process auths to control process with user cred, or whatever it would normally use, and then token is used to determine data transfer details.
- parallelism would always be 1 and multiple connections would only be supported via stripes.

Problems:
- parallel streams are very common, so now every connection takes N times as many resources as before.
- new clients and new servers would be needed at all ends for this to work.  hurts when source server requires this but dest server is not updated.

Possible fixes for problem #2:
- Must require new client. have client send session id to both ends via delegated proxy.  new server would read that id from data channel auth to know where the data belongs.  The trick is in manipulating delegated cred to handle session id.  There is a trouble in that for any connection received on 2811, GridFTP banner message is sent. When the data connection from an old server get banner, it will die.  But this can be solved by having separate port for control and data (2811 for control and 2812 for data).
- No need for new client. Ports 0-1023 are reserved and those values will not be send in the passive response by the existing servers. Single/dual port GridFTP server can use 0 -1023 as tokens and send it in the passive response. When the active server receives 0-1023 for port, it should know that this value is token and the server is actually is listening on 2811/2812. The problem is this will work only if there is new server on both ends.

Actual network reservation can be done either by one of the servers involved in the transfer or by the client, although it makes more sense to do it in the client end. Irrespective of where in the framework this interaction is implemented, it makes sense to have the GridFTP client control whether the network reservation needs to be done or not.

Network reservation integration on the GridFTP server: A new command to reserve resources (RSRV) has to be added to protocol. This command can be used to reserve network bandwidth as well as end system resources such as memory. GridFTP client has to end this command before the data channel connection is formed. It has to provide information on the resource requirement such as amount of bandwidth or memory required, duration etc.

GridFTP client globus-url-copy: GridFTP protocol does not let the data channel connection map {source host, source port, destination host, destination port} to be known to the client or the receiving server. Striping and parallel TCP connections introduce added potential complications and limit the client's ability to speculate the limited connection map {source host, destination host}. The control channel hosts and data channel hosts are not same for striped (or multi-node) transfers. So, protocol changes are needed to provide client with the data mover information to make the reservation. A new command (BIND) that lets the client to determine the host, port information for the source data movers is needed. Client can determine the host, port information for the destination data movers using the current GridFTP protocol.

The goal is to design the new commands and implement them.

OSCARS is the network reservation system for ESnet. The network reservation component in GridFTP will be developed in a modular fashion so that it will be able to interact with multiple end-to-end reservation systems (e.g., OSCARS, TeraPaths, DRAGON) and can be used in many environments; DOE labs using ESNet, science labs using Internet II etc.

The goal here is to integrate with OSCARS.

Assist with the creation of XIO modules for emerging alternative protocols such as RDMAoE

Create OS-native GridFTP-Lite Distribution with the goal to make it available as part of standard OS distributions and be available to a wide range of users.

Develop capabilities in GridFTP to enable coarse-grain allocation of system resources such as CPU and memory for data transfers

when user submits a job with proxy lifetime < job lifetime, its record never appears in audit. Unless user restarts the job to check job status on completion, in which case there will be a record.
It seems that when job manager dies, job is not audited. job manager can also die when gt machine is restarted.

While holding the manager lock, register_job_id is called.  Eventually, one of the child functions calls the manager lock again, resulting in deadlock.

If you do an "lsof" on a globus-job-manager, you'll notice that it holds open file handles pointing at the globus-gatekeeper.log.

We should have globus-gatekeeper.log opened with FD_CLOEXEC so the job-manager doesn't inherit it.

In some call paths to restart a job, the **old_job_request object may be NULL.  There is an unchecked dereference, resulting in a segfault.

Note that, based on the code, I'm taking an educated guess of the correct error code.  Would be useful to have an expert review.

Adding this bugzilla entry to jira for tracking.

https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=6768

when a Condor job is in the held state, GRAM should report the job's status as SUSPENDED, since it certainly isn't running.

The softenv implementation for fork only occurs in the fork-starter code path, so it won't work for job managers where the SEG is not used for fork.

We have an OSG site who is complaining about segfaults regularly occurring in globus-job-manager.  See https://ticket.grid.iu.edu/goc/12056 (ticket includes a corefile).

Looking through the core, it segfaults at globus_gram_job_manager_contact.c:1471.  The context pointer from globus_fifo_peek has value 0x20.  I perused the source and didn't find any obvious way for this to occur, making me think there is a memory management issue.

The globus-job-manager-script.pl program doesn't pass the $job_description to the run_command() subroutine, so if $FILTER_COMMAND is non-NULL, it will not get the actual executable and arguments from job description passed to it.

Investigating a user report, it seems likely that the SEG (at least the PBS SEG module) is capably of hitting deadlock states which prevent jobs from advancing at all from any job states expected by SEG events. This should be investigated and fixed.

The RSL substitution evaluation functions do not provide any context information when they fail. Their interface is defined to return only a success or failure value with no other error information included. At the minimum, it would be helpful to be able to have an API function to get info about the last evaluation error for the current thread.

The globus XIO close call can deadlock - it requires two free threads to complete.

The attached patch makes the code give up on close after 60s.  It will possibly leak some resources, or close things immaturely (if there are pending requests), but it beats a deadlock!

Vladimir Mencl  reports that having the job manager SEG module configured to parse PBS logs causes it to go into a cycle of high CPU and memory use. This should be detected better and treated as a misconfiguration failure if possible.

In threaded mode, calling globus_gram_job_manager_destroy requires multiple threads (as it calls globus_io_close, which is blocks on a second callback).  For fork'd g-j-m, no other threads exist.

The attached patch simply doesn't call destroy for these processes.

The fork LRM implementation has code to handle the softenv RSL attribute (in some cases) but the attribute is not defined in the fork.rvf file so it can't be used by default without some tricks.

This is an IGE project internal ticket that project team has decided to forward to Globus:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The return code from the payload should be returned by Globus GRAM tools (like globus-job-status or globus-job-submit) - or there should be some way to easily and uniformly obtain this return code.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Can you comment if such functionality will/can be implemented?

OSG has some patches to GRAM's condor LRM script to avoid using NFS between the service node and compute nodes. We should investigate these patches and get the equivalent functionality into the LRM scripts we distribute.

A GRAM log files can be difficult to read and find the important error or debug information inside.  A tool could be written to help a user find the important information inside a gram log files.

Tasks:
        - Create and review examples with potential users of this program.
        - decide on the format and options required
        - implement program

Every so often, globus-job-manager may deadlock or have threads die.

In such a case, we ought to have a watchdog timer in the main thread that will cause the globus-job-manager to die if it hasn't heard from the Globus callback system in awhile.

To improve scalability, performance and efficiency for a single client processing 1000s to 100,000s of jobs, GRAM could be enhanced to allow a client to send a vector of operations.  Currently, for a client to process 1000 jobs, that will require each job to perform a set of operations (round trips) from client to service.  It would be much more efficient for the client to construct a vector of operations and enhance both the client and service to be able to process a vector.  The operations job submissions, 2-phase commits, job status queries, job cancels, subscribing for notifications, etc.

Work closely with the Condor-G team (and others) to make sure the new vector operations can be used by them to improve scalability, performance and efficiency.

From a user:

Why installing and configuring a PBS based CE, I noticed that the globus-gram-job-manager-pbs-setup-seg rpm has a dependency on the torque-server.  However, it looks like the dependency is there only because the SEG uses the PBS accounting logs.  However, I'd argue that the typical resource is exporting these files to the CE using nfs and would not have the torque server installed on the CE.  Since globus gets the location of the pbs logs from the /etc/globus/globus-pbs.conf file, can we drop this dependency and just document what steps the admin needs to take in order to use the SEG?

What do you think? Is the user right? Should we change it?

The RPMs generated using the nordugrid patches build everything threaded. There is a suspected bug in the SEG which causes deadlock [GRAM-139]. The job manager and clients have not been tested with threads as part of the release process, so there may be other issues. The task here is to compile and run the tests with threads and try to locate the problems that occur using the existing tests with threaded clients and service implementations. As issues are discovered other bug issues should be added to jira. The tests runs should include the protocol, client, and job manager test suites built with threads, configured with fork and some other LRM, in both cases with and without SEG enabled.

I'm getting the following warning/error from GRAM:

ts=2012-02-26T10:18:27.325249Z id=6819 event=gram.send_job.end level=WARN status=-3 errno=2 msg="Error creating datagram socket" reason="No such file or directory"

However, I don't know what what socket it is trying to create.  The log message should be extended to include this in order to help debugging.

There are many old bugs in bugzilla that probably no longer apply.  Joe and Stu will review and cleanup the bugzilla gram bugs.

We currently do not have a profile of the execution of the GRAM5 Job Manager. Having such a profile would help us focus on performance optimizations which will most improve GRAM5. I think that we can generate some high-level data from the CEDPS-style logging implementation, but there may be some events which are not logged, or which require additional start or end messages. Otherwise, we will have to add some other metrics collection code to record what is occurring in GRAM5.

Once we have a way to collect this information, we should generate performance profiles for various job loads so that we can have a better view of the performance picture and how it relates to service scalability.

Fedora has replaced system v-style init scripts with systemd http://freedesktop.org/wiki/Software/systemd which has its own way of doing dependency-ordered process initialization. The GRAM5 services (globus-scheduler-event-generator and globus-gatekeeper) would need to have some glue written to adapt to this startup method.

Create a comparison document for GRAM5, comparing features with GRAM4 and CREAM, similar to that in http://www.globus.org/alliance/publications/papers/TG07-GRAM-comparison-final.pdf

The GRAM5 docs refer to the condor seg, but that is no longer used in 5.0.2. That is mentioned in the changes page, but the main documentation needs to be cleaned up as well.

One incredibly useful mechanism for debugging in Java is that, when you send the process SIGQUIT, it will dump stack trace of all threads and memory statistics to stdout.

We should have a similar tool for globus-job-manager.  I propose we catch SIGQUIT and dump to the log all items in the request hash and a few pertinent pieces of information (at least job and jobmanager state).

This would help immensely in debugging GRAM-319.

The LRM setup packages treat the perl code installed in $GLOBUS_LOCATION/lib/perl as data that needs to be generated at runtime. It would simplify external packaging if the configuration could be separated from the LRM perl code, and make it have reasonable defaults if the configuration values are not present.

When the SEG was first designed, it was intended to be run by a GT4 container to process LRM events. One instance ran per container per LRM to process log events.

In GRAM2 and GRAM5, we didn't want to run a SEG per user per LRM, so we wrote a script "globus-job-manager-event-generator" that executes the (potentially privileged) globus-scheduler-event-generator program and writes its output to log files in a compact, LRM-independent format. One globus-job-manager-event-generator program is run per LRM. The job manager uses the  seg_job_manager_module to parse the logs generated by this program.

As a result of this architectural shift, the globus-scheduler-event-generator program is no used independently of the globus-job-manager-event-generator script. The functionality of these can be combined into a program that behaves like the latter.

Another thing to add is an init script for the SEG program to allow it to be started at boot time for all installed LRMs. By default, it can search $libdir for libglobus_seg_*.la and start a SEG process for each, with a configuration file to explicitly set which SEGs to start and what user to run those as.

Currently the fork LRM is implemented in the following packages:
globus_gram_job_manager_setup_fork
globus_scheduler_event_generator_fork
globus_scheduler_event_generator_fork_setup
globus_scheduler_event_generator_fork_test
globus_fork_starter
globus_scheduler_provider_setup_fork
globus_wsrf_gram_service_java_setup_fork

The last two should be removed from the packaging metadata and "cvs rm"ed as they are gram4/mds4 specfic. The first 5 should be combined into the globus_gram_job_manager_fork package.  This package would provide
    lib/perl/Globus/GRAM/JobManager/fork.pm
    lib/libglobus_seg_fork_$(GLOBUS_FLAVOR_NAME).la
    libexec/globus-fork-starter
    etc/globus-fork.conf
    etc/grid-services/jobmanager-fork
    sbin/globus-gram-setup-fork

The fork.pm file will be made a distributed file, instead of shipping fork.in and a script to transform it to fork.pm. The config file will contain the configurable items and the globus-gram-setup-fork program will optionally do the probes. That can be used by either a gpt setup package or the rpm/deb postinstall phase. The default globus-fork.conf file should look something like:

# Path to the fork SEG log file. This is used to tell the fork starter where to
# write log entries and the fork seg module where to read them from.
#
# log_path=${localstatedir}/globus-fork.log

# Path to the mpiexec command used to launch mpi2 jobs
# mpiexec=/usr/bin/mpiexec

# Path to the mpirun command used to run older mpi jobs
# mpirun=/usr/bin/mpirun

# Path to the softenv installation used to set up the job environment
# softenv_dir=

The GRAM Condor LRM packages are split among
globus_gram_job_manager_setup_condor
globus_wsrf_gram_service_java_setup_condor
globus_scheduler_provider_setup_condor

The latter two should be removed from the packaging list and "cvs rm"ed. The first should be changed into a non-setup package that provides the following:
    lib/perl/Globus/GRAM/JobManager/condor.pm
    etc/grid-services/jobmanager-condor
    etc/globus-condor.conf
    sbin/globus-gram-setup-condor

(Note that the condor SEG has been removed as of 5.0.2 and replaced by code in
the job manager to process per-job condor logs)

The perl module will, by default, look in the system default path for condor tools  (to work with a natively-packaged condor).  That can be
overriden by values in etc/globus-condor.conf (which by default will contain only comments about what parameters are valid and their default values).

The globus-gram-setup-condor script will probe for condor programs and modify the globus-gram-condor.conf file.  A separate GPT setup package can be created which just runs that at postinstall time. For native packages, this can be included in the postinstall rules of the rpm/deb.

I think we should remove condor-os and condor-arch from the default grid-services entry; we can easily compute that information (it's derived from
uname()) in the condor.pm module in place of doing so at postinstall time. We can ahve overrides for that in the globus-condor.conf file. Some admins have asked for more flexibility in those values so that, for example, x86 jobs can be submitted to both INTEL and X86_64 machines. We can accomplish that by defining a multi-value format for the config file.

Example configuration file:
#
# Path to the condor_submit executable
# condor_submit=/usr/bin/condor_submit

# Path to the condor_rm executable
# condor_rm=/usr/bin/condor_rm

# Path to the condor configuration file
# condor_config=/etc/condor/condor_config

# Default CondorOS requirement
# condor_os=@CONDOR_OS@

# Default CondorArch requirement
# condor_arch=@CONDOR_ARCH@

# Do file existance checking on jobs in the standard universe. If set to no,
# then jobs which refer to files which do not exist will exit with ambiguous
# error messages. However, if the execution file system is not the same as
# the submit machine's file system, this may cause jobs to fail which would
# run otherwise
# check_vanilla_files=yes

# Path to a script to launch an mpi job in Condor. If set to no, then
# MPI jobs will be rejected
# mpi_script=no

The PBS LRM is implemented across the following files:
globus_gram_job_manager_setup_pbs
globus_wsrf_gram_service_java_setup_pbs
globus_scheduler_event_generator_pbs
globus_scheduler_event_generator_pbs_setup
globus_scheduler_event_generator_pbs_test
globus_scheduler_provider_setup_pbs

The wsrf and scheduler_provider setup packages can be removed from CVS as they aren't needed for GRAM5.
The others can be combined into a single package that provides:

    lib/perl/Globus/GRAM/JobManager/pbs.pm
    lib/libglobus_seg_pbs_$(GLOBUS_FLAVOR_NAME).la
    etc/globus-pbs.conf
    etc/grid-services/jobmanager-pbs
    sbin/globus-gram-setup-pbs

The globus-gram-setup-pbs probes for pbs tools and offers command-line options for the other parameters, modifying the globus-pbs.conf file. The pbs.pm file will not be autoconf substituted, but instead will read the config file for values. The setup program can be run as a setup package or via native packaging postinstall support in the rpm/deb.

The default configuration file should look something like this:
# Path to mpiexec program to launch mpi2 tasks
# mpiexec=/usr/bin/mpiexec

# Path to mpirun program to launch older-style mpi tasks
# mpirun=/usr/bin/mpirun

# Path to qsub program to submit a job to the LRM
# qsub=/usr/bin/qsub

# Path to qstat program to poll a job's status
# qstat=/usr/bin/qstat

# Path to the qdel program to cancel a job
# qdel=/usr/bin/qdel

# Flag indicating whether PBS is configured as a cluster or simple SMP machine
# cluster=yes

# Number of compute elements per schedulable nodes
# cpu_per_node=1

# Remote shell program to start executables on different nodes in the
# $PBS_NODEFILE
# remote_shell=/usr/bin/ssh

# Path to the softenv installation used to set up the job environment
# softenv_dir=

The LSF LRM implementation consists of the following packages:
- globus_gram_job_manager_setup_lsf
- globus_wsrf_gram_service_java_setup_lsf
- globus_scheduler_event_generator_lsf
- globus_scheduler_event_generator_lsf_setup
- globus_scheduler_event_generator_lsf_test
- globus_scheduler_provider_setup_lsf

The wsrf and provider packages can be removed from CVS. The others can be combined into a gram lsf package that provides:

    lib/perl/Globus/GRAM/JobManager/lsf.pm
    lib/libglobus_seg_lsf_$(GLOBUS_FLAVOR_NAME).la
    etc/globus-lsf.conf
    etc/grid-services/jobmanager-lsf
    sbin/globus-gram-setup-lsf

The substituions previously done to lsf.in can be done instead to globus-lsf.conf, so that the deployed script isn't modified and all of the configuration parameters are clearly described. The globus-gram-setup-lsf program can for lsf tools and offer command-line options for the other parameters.  A separate GPT setup package will be created which just runs that at postinstall time. For native packages, this can be included in the postinstall rules of the rpm/deb.

Example Configuration file:
# Path to the LSF shell profile
# lsf_profile=/opt/lsf/conf/profile.lsf

# Path to mpirun program to launch older-style mpi tasks
# mpirun=/usr/bin/mpirun

# Path to bhist program to poll a job's status
# bhist=. $lsf_profile && bhist

# Path to bsub program to submit an LSF job
# bsub=. $lsf_profile && bsub

# Path to the bjobs program to get information about a job
# bjobs=. $lsf_profile && bjobs

# Path to the bkill program to cancel a job
# bkill=. $lsf_profile && bkill

Update the SGE LRM package to be less dependent on GPT setup scripts. Move configuration code out of the LRM perl module into a configuration file. Combine the SEG and LRM modules into a single source package.

A high-level diagram is needed in the "approach" documentation to help people understand the GRAM5 architecture.

The first time the globus-job-manager-event-generator is run, it will read all existing LRM logs and write SEG events for them. In the case of a heavily used system or one with a long history, this can take a very long time and use much CPU. It might make sense to either add a command-line option to skip the historical events or base it off of the time when the software was installed, with the assumption that no events relevant to a new GRAM installation will occur before GRAM is installed.

An option for globusrun and globus-job-run like the -dbg option for gridftp, which turns on higher levels of debugging statements.

sge.pm is not supporting modules on TACC ranger. The MPI LD_LIBRARY_PATH and other variables are not getting loaded and MPI jobs do not run through gram5 unless the user environment is explicitly added to the user login environment. Can the SGE.pm be changed to support modules as a permanent solution?

Thanks,

In the how OSG uses Globus doc (may 2009), they request that GRAM provide support for gram-level prologue and epilogue script execution for mpi jobs (but it could be for any job)

GRAM4 has softenv extensions but GRAM5 does not. Nanohub has requested this feature in GRAM5

The GRAM5 code by default will run up to 5 perl scripts per job manager simultaneously. We should probably investigate whether having more than 1 is worthwhile, and if so, make it a tunable parameter from the job manager configuration file.

The GRAM5 code uses a common parent directory for creation of the job-specific directories, which contain proxy, stdout, stderr, and lrm-specific files. On some filesystems (ext[234]), there is a limit to the number of hard links a file (in this case, the links ".." in the subdirs) can have, so after creating a large number of jobs, errors like the following occur

GRAM Job submission failed because mkdir failed: /home/bamboo/.globus/job/ip-10-190-201-161/16217855108623346281.6322278966767028136: Too many links (error code 22)

It is not easy to figure out from our developer doc how to write a reliable gram client.  Condor-G has done it, but I don't think many others have.  The doc should be improved to describe how to write a reliable multi-threaded gram client using GRAM5.

More helpful error codes to assist in debugging would be a big help.  Perhaps a larger number of more specific errors could be used, or errors could include stack-trace-esque information that could be used to help debug.

For any LRMs that can optionally use SEG or poll for LRM job monitoring, if SEG is configured, but SEG events are not being received quickly enough (indicating problems / misconfiguration) the GRAM service should fallback to using poll to get the job status.  Additionally, an error or warning should be output to notify the admin about the problem.

To be considered in the 5.2 repackaging work, make the job audit logging easy to deploy.  Ideally installing a single package.

This was a significant issue last year when we were trying to understand what clients were out there using gram.

It is similar to GRIDFTP-60.

GRAM5 security protocol requires GSI delegation.  Because of this standard SSL implementations cannot be used.  Delegation could be factored out of the GRAM protocol and reimplemented in the application layer.

Raminderjeet Singh has requested support for the "name" RSL attribute for the SGE LRM adapter. It currently is only implemented in the PBS adapter.

The current throughput tester program could be simplified by stripping out the GRAM4-isms.

We probably want throughput tester programs written in C and Java.  These programs should be doc'd well in order to use them as GRAM5 client code examples.

OSG asks that GRAM2 add support for a "managed fork" service.  Today, condor is used.  But the requirements may be simple enough for improvements to be made to Fork in order to avoid the condor dependency.

Hi Stu,

I'm helping deploy some infrastructure at UNC Chapel Hill, and have run
into some GRAM5 (5.0.3) issues I would like help with.

The site is running LSF 7, and I have already made some changes to
lsf.pm (diff attached). I suspect that the your lsf.pm was developed
against LSF 6, and that could explain the changes necessary. I think I
have a good handle on this problem, and we might even expand on the
changes ones we get past the two problems below.

We are running without SEG. SEG testing will be done once we know that
the basic mode works. The client is Condor-G.

Problem 1: Sometimes, maybe every 100 jobs or so, a job status becomes
"stuck". For example, globus-job-status will keep on returning ACTIVE
forever. It seems like the poll method in lsf.pm is never reached, so I
assume that the status is just picked up from the state file. I'm
wondering why it is not updated, and why poll() is not called in this
case. Killing the globus-job-manager process, and then running
globus-job-status again will return DONE.

Problem 2: Possibly related to high load and/or NFS mounted home
directories, but sometimes we end up with errors about not being able to
open lock files. For example:

ts=2011-08-10T17:06:16.943530Z id=12875 event=gram.state_file_read.end
level=ERROR gramid=/16145785936629637586/6793341820049045889/
status=-158
path=/usr/local/globus-5.0.3/tmp/gram_job_state/job.gnet641.its.unc.edu.16145785936629637586.6793341820049045889
msg="Error opening job lock file" errno=2 reason="No such file or directory"

I don't have a good idea for why this is happening, but I feel like the
handling of the state files and lock files could probably be made a
little bit more robust.

Any help appreciated,

--
Mats Rynge
USC/ISI - Pegasus Team 

OSG raised the security issue recently with file staging in Condor-G using GRAM.  Condor-G uses a long running GAHP service for submitting and processing a users jobs.  The GAHP server starts a GASS server for file staging.  If a user's proxy is stolen, then it could be used to push/pull files from/to the GAHP's GASS server.

2 ideas proposed by OSG:
  a) change to a model where files are pushed to the GRAM service
  b) restrict the files/dirs available to a GASS server

Sites add functionality to the GRAM LRM submission script.  Typically, a patch
is written and maintained.  It is applied to each release / install.  This
method can be problematic since the patch may not apply cleanly depending on
the changes made in the script.  One possibility would be to define a set of
callout(s) that will be run in the script.  Sites can then implement a callout
function instead of a patch.  This would avoid the fragile patch method.

When David Carver was recently deploying and testing GRAM5, he could have used better error messages.  He used this table to help find solutions:  http://www.globus.org/toolkit/docs/5.0/5.0.1/execution/gram5/user/#gram5-error-codes
From that he requested the "possible solution" info be added to the error output of globusrun.

Here is an example of an error message you get from GRAM in GT 5.0.1:
-------
$ globusrun -s -r never-1.ci.uchicago.edu "&(executable=/bin/notThere)"
GRAM Job failed because the executable does not exist (error code 5)
-------

David's recommendation for the error output is this:
-------
$ globusrun -s -r never-1.ci.uchicago.edu "&(executable=/bin/notThere)"

GRAM Job failed because the executable does not exist
Error Code: 5
Reason: the executable "/bin/notThere" does not exist
Possible Solution: Check that the RSL executable attribute refers to an executable that exists on the target system.
-------

Looks like a good suggestion to me.

ATLAS (and others) want to be able to cluster a set of GRAM2 services in a HA setup to provide greater scalability and reliability.

This is from Ian Stokes-Rees (OSG) by way of Alain Roy:

We are struggling with a challenge presented by GASS cache.  A common mode for us to work in is:

1. User develops job script ~/osg/work/run.sh
2. User submits 20 jobs Monday which use run.sh as executable
3. User looks at results Tuesday and tweaks run.sh
4. User submits 20 more jobs Tuesday which use run.sh as executable.

We have just discovered that the GASS cache (at least at many sites with their current setting) will result in the FIRST "run.sh" script from Monday being used instead of the SECOND "run.sh" script edited on Tuesday.  Not surprisingly, this is undesired behavior.

The GRAM client API uses the io_compat API instead of XIO directly.  In order to set timeouts, the GRAM client API needs to use XIO directly.  This should be changed in GRAM5.

Jaime Frey wrote:
Today, I investigated a problem observed by Igor Sfiligoi (CC'd) that I was hoping you could comment on. He's submitting Condor-G glideins to numerous sites. In the past couple days, all gram commands started returning connection failures, though the sites were functioning normally. His gahp_server had run out of fds (fd limit of 1024). All of the fds were tied up in established connections to 20 job-managers at one site, up to 100 connections to each job-manager. The number of connections to each job-manager was roughly equal to the number of gram commands the gahp had sent to that job-manager. The gahp was trying to read data off of the connections. When I killed the jobmanagers, all of the fds were quickly closed.
The gahp server was linked with Globus 4.0.5 libraries. The remote site appears to be running a late-2.4.x release of Globus. My question is why didn't the Globus libraries in the gahp server timeout on the connections after several minutes? Are timeouts off be default in globus_xio?

The tests which refresh credentials in the globus_gram_client_test package sometimes fail if the job terminates before the refresh completes. I've marked this as a TODO test in 5.0.5 so that it won't affect bamboo build results.

I find myself looking up specific function calls, and the current page does not make this easy. A search field on would be nice, or an index containing all functions that I could Ctrl-f and link to the description.

The build appears to abort compilation when attempting to pull in ssl. This URL references the full output:
http://nmi-s003.cs.wisc.edu/nmi/index.php?page=results/runDetails&runid=231648&MetronomeSessID=d3h57hrb1ejv8p3hiiceuqc535&opt_user=wmihalo

This applies to cred_get_issuer_name as well. There is a comment:

/* ToDo: This logic needs fixing. The issuer_name is passed up and is
             freed by the caller - but it must be freed with OPENSSL_free(),
             not free() and the caller cant be expected to know that */

But the doc header (and generated API doc) does not say anything about using OPENSSL_free. cred_get_X509_*_name do document their free requirements. I imagine the behavior is relied on by many applications at this point, so just documenting the requirement seems the best course.

Some of the globus_io test cases fail or hang because of a bug in the XIO GSI driver. The problem is occurring in the handling of GLOBUS_XIO_GSI_ACCEPT_DELEGATION or GLOBUS_XIO_GSI_REGISTER_ACCEPT_DELEGATION cntl implementations. If the read registered to get the accept_sec_context tokens reads (all or part of) the accept_delegation   token, it will be ignored by the accept_sec_context(), but then the cntl implementation will try to register another read to get that token. If the token has already been completely read, it will cause the server to hang in the select waiting for the token, if the token has been partially read, it could cause the server to get an incorrect token length. The solution is probably to use the same buffering code used in the init/accept sec context code so that tokens are properly read. I've not seen anything that uses this except the globus_io_accept_delegation and globus_io_register_accept_delegation code and tests for that code.

not even sure if this is the right project.. but building the 5.2 toolkit from source, configure --libdir=X is ignored on linux x64 and keeps getting set to prefix/lib64.  I dont want lib64.

5.2 install from source, prefix to /usr/local/foo.

The installed sbin/SXXsshd has two problems.

. ${GLOBUS_LOCATION}/libexec/globus-script-initializer  # 

The GSSAPI Extensions documents (http://www.ggf.org/documents/GFD.24.pdf  and older version at http://www.ggf.org/security/gsi/draft-ggf-gss-extensions-07.pdf) define id-gss-ext-context-opts-disallow-encryption and GSS_DISALLOW_ENCRYPTION respectively. GSI C defines GSS_DISALLOW_ENCRYPTION and when specified sets the GSS_I_DISALLOW_ENCRYPTION flag on the context, but the flag is NOT enforced.

It is recommended that either the flag be enforced or GSI C modified to return a "Not Implemented" error when GSS_DISALLOW_ENCRYPTION is specified by applications. The latter is probably preferable.

MyProxy SRPM in 5.1.1 fails to rebuild.  It appears that when we run %check, it makes a call to "grid-proxy-init", which is not brought in by the BuildRequires:

Not sure of the correct Globus project to file this under, but hopefully this gets to the right place.

The readline header file has been detected but the readline library -lreadline has not been (it happens when GT with 64-bit flavor is built, and the readline is 32-bit). However, the macro -DHAVE_READLINE=1 is set on. It looks like that the configure script decided that the readline library is also availbale. It causes problems because symbols supposed to be defined in the readline library cannot be found.

[gt-installer]$ ./configure --prefix=/home/condor/execute/dir_4041/userdir/install --with-flavor=gcc64dbg --with-buildopts='-verbose' LDFLAGS=-L/usr/local/lib



CPP='/prereq/gcc-3.4.3/bin/gcc -E'; export CPP; CPPFLAGS=' -I/home/condor/execute/dir_4041/userdir/install/include -I/home/condor/execute/dir_4041/userdir/install/include/gcc64dbgpthr'; export CPPFLAGS; CFLAGS='-g -D_REENTRANT -std=gnu99 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -m64 -D_REENTRANT -Wall'; export CFLAGS; LDFLAGS='-L/usr/local/lib -L/home/condor/execute/dir_4041/userdir/install/lib -m64 '; export LDFLAGS; LIBS='-lsocket -lnsl -lpthread -lposix4'; export LIBS; CXX='/prereq/gcc-3.4.3/bin/g++'; export CXX; CXXCPP='/prereq/gcc-3.4.3/bin/g++ -E'; export CXXCPP; CXXFLAGS='-g -D_REENTRANT -m64 '; export CXXFLAGS; AR='/prereq/binutils-2.21/bin/ar'; export AR; ARFLAGS='ruv'; export ARFLAGS; RANLIB='/prereq/binutils-2.21/bin/ranlib'; export RANLIB; NM='/prereq/binutils-2.21/bin/nm -B'; export NM; CC='/prereq/gcc-3.4.3/bin/gcc'; export CC; ./configure --prefix=/home/condor/execute/dir_4041/userdir/install --enable-threadsafe --disable-tcl



checking for readline in -lreadline... no
checking readline.h usability... no
checking readline.h presence...
no checking for readline.h... no
checking for /usr/include/readline.h... no
checking for /usr/include/readline/readline.h... no
checking for /usr/local/include/readline.h... no
checking for /usr/local/include/readline/readline.h... yes



creating libsqlite3_gcc64dbgpthr.la
(cd .libs && rm -f libsqlite3_gcc64dbgpthr.la && ln -s ../libsqlite3_gcc64dbgpthr.la libsqlite3_gcc64dbgpthr.la)
./libtool --mode=link /prereq/gcc-3.4.3/bin/gcc -g -D_REENTRANT -std=gnu99 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -m64 -D_REENTRANT -Wall -I. -I./src -DNDEBUG -DTHREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DHAVE_READLINE=1 -I/usr/local/include/readline -lpthread \
   -o sqlite3 ./src/shell.c libsqlite3_gcc64dbgpthr.la \
   -lcurses -lrt /prereq/gcc-3.4.3/bin/gcc -g -D_REENTRANT -std=gnu99 -D_XOPEN_SOURCE=600 -D__EXTENSIONS__ -m64 -D_REENTRANT -Wall -I. -I./src -DNDEBUG -DTHREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DHAVE_READLINE=1 -I/usr/local/include/readline -o .libs/sqlite3 ./src/shell.c ./.libs/libsqlite3_gcc64dbgpthr.so -lpthread -lcurses -lrt -R/home/condor/execute/dir_4041/userdir/install/lib Undefined first referenced symbol in file
write_history /home/condor/execute/dir_4041/ccdjLdc4.o
stifle_history /home/condor/execute/dir_4041/ccdjLdc4.o
read_history /home/condor/execute/dir_4041/ccdjLdc4.o
readline /home/condor/execute/dir_4041/ccdjLdc4.o
add_history /home/condor/execute/dir_4041/ccdjLdc4.o
ld: fatal: Symbol referencing errors. No output written to .libs/sqlite3
collect2: ld returned 1 exit status
make[1]: *** [sqlite3] Error 1
make[1]: Leaving directory `/home/condor/execute/dir_4041/userdir/gt-installer/source-trees-thr/database/c/sqlite/sqlite-3.3.17'
ERROR: Build has failed
make: *** [globus_database_sqlite-thr-compile] Error 9

When the globus-usage-uploader hits an error trying to upload a set of data to the database, it aborts the current transaction. This has the side-effect of getting some of its locally-cached state out of sync, so that some things in future uploads which refer to the same host or service might end up with ids which aren't in the database and cause further errors. I think this is why when the usage stats uploader is behind and tries to load multiple files of packets to the database, they tend to be all-or-nothing failures. I think it should be possible to flush some of the caches when a transaction is aborted and keep things working (though somewhat slow). It might be better to separate the transactions into smaller chunks and then write failed chunks of packets to an error file instead of doing it at a full hour's worth of data level. In any case, data is not lost in this case, but requires some intervention to get into the database.

The i18n rules in the installer's makefile for 5.0.0 and 5.0.1 don't work, as the i18n code and bundle aren't included in the installer. Either the rules should be eliminated or the i18n code should be included in the installer.

When installing myproxy, a few dependencies are not listed as requirements.  In order to install myproxy, we need to install the following packages explicitly:

libglobus-usage0 libglobus-gss-assist3 globus-proxy-utils myproxy

step by step instructions for creating source RPMs and binary RPMs.  Same for Debian packages.

I tried to re-use a globus_gsi_callback_data_t when making calls to globus_gsi_cred_verify_cert_chain, and discovered that it consumed ~100kb per call until _destroy was called.This was surprising behavior to me as a new GT user. I am also re-using a globus_gsi_cred_handle_t to load different certificates, which does work.

I assume that callback_data is designed to be passed down a call stack, so the it can't return an error on attempted re-use. I think it would be helpful to add something to the callback_data_init/destroy API docs along these lines:

The callback_data will grow every time it is passed to a GT function; it is designed to be passed down a call stall, but it should be destroyed after each top level call is made.

This is an internal ticket of IGE project:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hello,

IPv6 is becoming more and more an issue everywhere. Can Globus work with IPv6? This is more a question than a request. However, if the answer is no, it can not work with IPv6, then the request will be to upport IPv6 inGlobus.

Thanks!
Helmut
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Here is a request from Oscar Koeroo 

Making the "gsi-authz.conf" file use the order in which the
"globus_mapping" and "globus_authorization" statements are listed. Top
first, followed by another one. At the moment we have to push both the
call-outs to our LCAS authorization and LCMAPS mapping frameworks in
the "globus_mapping" specific call-out as it does not fit any of our
deployment scenarios to perform the mapping process before an
authorization call-out. We procure poolaccounts or other state-full
system resources at the mapping sequences which should not be procured
when the authorization failed.

InCommon is moving to self-signed X.509 certificates, and away from having their own CA.

https://spaces.internet2.edu/display/InCCollaborate/X.509+Certificates+in+Metadata

This is something we talked about doing in Globus years ago.  We should take a look at exactly what they are doing in InCommon, and consider adding support for that approach in Globus.  At a minimum, support would presumably including making sure our C and Java security libraries will correctly validate a self-signed cert, and that the gridmap file can refer to them.

This may be useful for usability.  For example, it might provide a means for zero-config deployment of GridFTP servers that can be used by Globus.org.

To be considered in the 5.2 repackaging effort, include IGTF CA certificate distribution - and bundle in fetch-crl

Implement the plan created in RIC-60

Oscar Koeroo will provide a snapshot of the LCAS and LCMAPS (L&L) authorization code.  Integrate the L&L authorization code into Globus Toolkit 5 releases.  Retain gridmap authorization as the default.  Add new configuration options to use L&L authorization instead of gridmap for GRAM, GridFTP, and GSI-OpenSSH.

It would be awesome to get a homebrew (http://mxcl.github.com/homebrew/) package for globus toolkit for those of us working on os x.

The fedora 17 release is scheduled for release in May. We should add support for this in its current state so that we can shake out any GT-related bugs prior to its release.

As described by Jim Basney:



The changes in GT 5.1.0 are causing some issues when I try to install
things to $GLOBUS_LOCATION using gpt-build.

For example, when I do 'make gsi-openssh install' in the
gt5.1.0-all-source-installer directory, I get things installed in:

  $GLOBUS_LOCATION/share/globus
  $GLOBUS_LOCATION/share/man

which I believe is due (at least in part) to the config.site file in the
gt5.1.0-all-source-installer directory setting
libexecdir='${datadir}/globus'. But then when I do 'gpt-build
gsi_openssh_bundle-5.3-src.tar.gz gcc32dbg' I get things installed in:

  $GLOBUS_LOCATION/libexec
  $GLOBUS_LOCATION/man

presumably because there's no config.site file in this case. So rather
than upgrading the gsi_openssh files I have installed, I end up with new
versions installed in $GL/libexec and $GL/man while the old versions in
$GL/share are untouched.

When I try 'gpt-build myproxy-5.4.tar.gz gcc32dbg' using a GT 5.1.0
$GLOBUS_LOCATION I get:

  ERROR: Flavor gcc32dbg has not been installed
  ERROR: Build has failed

then after I do

  ln -s $GLOBUS_LOCATION/share/globus/flavors \
        $GLOBUS_LOCATION/etc/globus_core

I get:

  $GLOBUS_LOCATION/libexec/globus-build-env-gcc32dbg.sh:
  No such file or directory
  ERROR: Build has failed

So I try:

  ln -s $GLOBUS_LOCATION/share/globus/globus-build-env-gcc32dbg.sh \
        $GLOBUS_LOCATION/libexec/globus-build-env-gcc32dbg.sh

and then get:

  $GLOBUS_LOCATION/sbin/libtool-gcc32dbg: No such file or directory

so it seems the current myproxy package isn't compatible with GT 5.1.0.

If I create a new myproxy GPT package from GT 5.1.0, then I get similar
errors when trying to install it using gpt-build to a GT 5.0.3
$GLOBUS_LOCATION.

Is this expected? Will it no longer be possible for me to release GPT
packages that can be installed via gpt-build into $GLOBUS_LOCATION for
the different supported GT versions because of GT 5.1.0 GPT changes?
I'll need to release different myproxy & gsi_openssh GPT package
versions for GT 5.0.x versus GT 5.1.x/5.2.x?

From Cristina:

Can you verify the delegation pieces displayed in this section of the GT Users' Guide (for GT5)?

http://www.globus.org/toolkit/docs/5.0/5.0.0/user/#gtuser-security

Some of the packages distributed in GT 5.0.x are just gpt-wrapped versions of external programs. It would be helpful for the native repackaging effort to drop those so that we can better meet native packaging policies and reduce our support overhead.

There are three main ways I see to do this:
* Do like we do with globus_openssl in GT 5.0.x where we have gpt packages that are just metadata wrappers around external libraries. The pro for this are that we'd not have to change metadata for dependent packages. The con is that we still have packages to deal with that don't really bring much value.
* Hand code configure.in rules to detect and use the external libraries in packages that need them. This is done in the globus_openssl package. We'd have to duplicate those configuration rules (which use pkg-config if present, falling back to environment variables falling back to configuration options).
* Add a prerequisite for pkg-config, and use its autoconf macro packages to handle external dependencies. The pro for this is that we'd get some decent handling of compile and link lines for most everything we depend on and the actual expression of dependencies is quite simple (something like PKG_CHECK_MODULES([LIBXML2], [libxml-2.0]) is all we'd need in the configuration scripts). The con is that we'd add another external dependency.

It would be helpful to have this work done in such a way that the top-level configuration script for the installer is able to detect missing dependencies prior to trying to build dependent packages. This may be hard to automate, but the number of external dependencies is rather small, so some hard-coding might be ok.

Debian has a tool "lintian" which checks debian packages for compliance with the debian policies. I've looked a little bit at it, but haven't figured out how to run it, but we should probably add that to the build system before 5.2 is released, so that the packages will be more useful to Mattias for inclusion in debian.

Myproxy is creating flavored libraries, and putting its headers into flavored directories:

Also, I notice that GT 5.1.0 puts myproxy.h in

  $GLOBUS_LOCATION/include/globus//myproxy.h

rather than

  $GLOBUS_LOCATION/include//myproxy.h

and most other GT 5.1.0 headers are in

  $GLOBUS_LOCATION/include/globus

(i.e., not in a  subdirectory). Should I be changing something
in the myproxy package to put headers in
$GLOBUS_LOCATION/include/globus? I notice that myproxy is also the only
package installing flavored libraries in GT 5.1.0, which is probably
related.

Implement a cron job on the NMI server at Wisconsin that will reliably run Solaris Build and Tests for the LIGO project and for other users that might be interested in GT on Solaris.

test-toolkit, which ships with all of the GT software, needs better documentation. It should be placed into production on the NMI server for TEST runs. We are currently running gram-gridftp-test-package for TEST runs on the NMI server and have discovered that some critical branches of the TEST runs complete "successfully" without making any runs. This needs to be fixed. We also need to better document this useful utility so that other people within the GT community can make better use of it.

For native packaging it would be helpful to create external dependencies for the packages we ship in GT5 but which don't contain any patches beyond packging-related ones. RLS depends on iodbc, sqlite, sqlite_odbc, and psql_odbc. It's not clear to me whether those are all needed at compile time or whether RLS is safe to be updated for this work.

Tasks:
* Determine feasibility of updating RLS (who can do this?)
* Remove dependencies on sqlite, sqlite_odbc, and psql_odbc from RLS server and setup packages
* Add code to detect libiodbc and generate appropriate compile and link lines to RLS build

The globus_core and gpt packages contain code to manage mpi flavoring of libraries and executables. Since we have removed nexus in GT5, there is no more code in GT that uses this information, and so I don't think we need to support these features any more.

Tasks:
* Remove flavor labels related to mpi from gpt
* Remove detection code for mpi from globus_core

Annette DeSchon has asked that the RLS test suite be added back to the GT distribution. It was included through GT 4.2.1.

The GPT build-time scripts do three major things.  They do the dependency checking to ensure that all compile time dependencies are met.  They convert the source metadata to binary metadata/filelists.  They assemble the linklines for the package from the dependent package linking information.  The first of these can be worked around by building in a context in which the dependency existence and ordering has been predetermined, such as the new installer model proposed in RIC-90.

Once each package outputs pkg-config metadata as well as GPT metadata, the linklines should be just as easily assembled using pkg-config as using the GPT script gpt_build_config (called from configure, as found in the GPT_INIT macro).

Altering the GPT_INIT macro to allow the use of pkg-config instead of gpt_build_config would open up the possibility of source builds that do not depend on the presence of an installed GPT.  Backwards compatible behavior could be maintained.

Conversion of source to binary GPT metadata could either be handled by an XSLT transformation, or simply ignored in the no-buildtime-GPT scenario.

Currently there are none. Some of the other tests likely exercise this library, but only certain use cases.

Debian 7 will add multiarch support, which allows different architecture versions of the same package to be installed simultaneously on a single system with using the emulation libraries or chroots, which is nice. It is pretty different than the LSB, however, so work will need to be done to support this in our packaging metadata. Ubuntu has support for this as well. I'm not sure if all of the GT dependencies have been updated to support this, so we may be able to wait until things like openssl and libxml2 are updated for multiarch.

Get this page: http://confluence.globus.org/display/GT/GT+5.2+release+process up to date to include info about the bamboo scripts, frag generator, etc.

Jars certainly can be packaged up natively.  The standard way of doing this is like this:

http://packages.debian.org/lenny/all/libcommons-beanutils-java/filelist

However, as a Java developer, I rarely find this kind of packaging useful.  I would suggest that the most useful way to distribute JGlobus Java client libraries is 1) simple tar'ed distros and 2) via the Globus maven repository.

---

I can help Eric, et al. with how to set up build process/distribution process to facilitate distributing the clients via these channels.  I have some template files which could be fairly easily altered or extended for this purpose.

-Tom

First reported here: https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=6148

There are some parts of the various packet tables which contain combinations of two data items (java style host/ip, list of service names). Since we do queries on that field, we end up splitting the data on the client side. In some cases we don't care much about what the data is, just the distribution of different hosts or different services. These make queries slow.

investigate what is needed to add support for SHA-2 credentials in the Globus Toolkit

From Oscar Koeroo 

Other differences are how logging output is pushed to a logfile or
syslog. There are differences experienced in that regard to slip-stream
it with the gatekeeper, gridftpd and gsi-opensshd logging facilities.
Ideally we'd like to hook our logging facility into that what has been
chosen for the service itself.

The Nordic Grid computing project that produces the ARC middleware (see www.knowarc.eu) has done some work to port and get some of the Globus common components into some of the common Linux distributions, notably Debian & Ubuntu. Matthias Ellert in Uppsala, Sweden who has been doing the work, here are samples.

Here is the list of source packages (GPT + 30 Globus packages):

http://packages.debian.org/source/sid/grid-packaging-tools
http://packages.debian.org/search?keywords=globus&searchon=sourcenames&suite=unstable&section=all

The packages are slowly migrating from unstable (sid) to testing
(squeeze) [currently GPT + 3 of the Globus packages]:

http://packages.debian.org/source/squeeze/grid-packaging-tools
http://packages.debian.org/search?keywords=globus&searchon=sourcenames&suite=testing&section=all

We should review what has been done here and commit any changes to the common components to facilitate this process of including them in Debian and Ubuntu.

The report generator java code is quite complicated for what it achieves, essentially creating and collating the relations on the client side instead of in the SQL. Part of this is because the database schemas are terrible, but also because the report generator code is pretty convoluted itself.  Joe has done some work on getting the reports to cache results in the database, but it needs more work before it can be used.

The globus simple ca can generate CA certificates, but doesn't have an option to regenerate a new CA certificate using the same private key that was previously used. If this were implemented, one could generate a new CA certificate after the original expires which would still validate the signatures of keys issued with the previous CA cert. Originally requested: http://lists.globus.org/pipermail/gt-user/2011-October/010126.html

This request comes from an OSG/Globus meeting...

Our view is that Globus support for GSI is incomplete--how do we handle CA certificates? CRLs? OSG now has tools to address these needs from outside of Globus.. That said, OSG would like to suggest that Globus revisit how CRLs work in GSI. Right now in the C implementation, a CA's CRL that has passed the "nextUpdate" is considered to be expired and no certificates issued by that CA can be accepted. However, if no CRL is present, authentication can proceed. We understand the Java GSI implementation is inconsistent with this. We realize that changing this behavior is not simple (it's not just a technical issue), but we suggest harmonization of the implementations.

It would be useful to have usage graphs that are always up-to-date.  Currently, the daily reports only provide numbers.  We could create a script that loads data from the metrics database into Google Docs, so that we always have up-to-date graphs.

RIC-265 had a leak in the gssapi that was showing up as a problem for OSG when running long-lived GRAM job managers. I did a valgrind run with some of the test cases and it showed a few other recurring leaks in some of the newer code.

When going through core files on my CE, I found the following abort relatively frequently:

(gdb) bt
#0  0x00000034ae630265 in raise () from /lib64/libc.so.6
#1  0x00000034ae631d10 in abort () from /lib64/libc.so.6
#2  0x00000000004125df in globus_gram_job_manager_request_free (request=0x1cee5cd0) at globus_gram_job_manager_request.c:1315
#3  0x000000000040c02c in globus_l_gram_job_manager_remove_reference_locked (manager=0x7fff33563290, key=0x1cd6d360 "/16217900475151569756/11840111202095113110/", reason=0x42c6fc "stop all jobs")
    at globus_gram_job_manager.c:1122
#4  0x000000000040c6c6 in globus_gram_job_manager_remove_reference (manager=0x7fff33563290, key=0x1cd6d360 "/16217900475151569756/11840111202095113110/", reason=0x42c6fc "stop all jobs") at globus_gram_job_manager.c:988
#5  0x000000000040c8f3 in globus_gram_job_manager_stop_all_jobs (manager=0x7fff33563290) at globus_gram_job_manager.c:2201
#6  0x00000034b361850e in globus_callback_space_poll_nothreads () from /usr/lib64/libglobus_common.so.0
#7  0x00000034b363846f in ?? () from /usr/lib64/libglobus_common.so.0
#8  0x0000000000409432 in main (argc=, argv=) at main.c:611

Sample core file attached.  Relevant RPMs (including debuginfo RPMs) are here: https://koji-hub.batlab.org/koji/buildinfo?buildID=1694

Looking at that line of code, it seems to be relatively harmless.  We could probably just log and move on - sysadmins get nervous when they see SIGABRT show up in the kernel logs.

SLES is used by XSEDE for Kraken and Nautilus. We don't yet generate RPMs for that platform. It has some differences from fedora and redhat in its package naming for dependencies, and uses a different tool zypper for downloading from a repository, so the spec files and build and test scripts will need some changes for that.

05-09 15:00:18.790 conn_send_cmd_v D0.14> SITE UTIME 20120504171508 ~/test5
05-09 15:00:18.790 conn_send_cmd_v D0.15> MDTM ~/test5


05-09 15:00:18.960 _conn_cmd_cb D0.14< 250 OK.
05-09 15:00:18.960 _conn_cmd_cb D0.15< 213 19700101010220

VDT had patches in place to make globus-url-copy use GET/PUT (GridFTP V2) by default, as well as a ftp client library patch to allow enabling GET/PUT by environment variable, for older apps that had no way to enable it via the api.  As some users of VDT transition to standard Globus releases, it would be good to add a solution for this.  It should be alright to enable it by default for hosts that support it.  If there is a good reason not to do that, I can add the environment variable to enable the feature.

We see about 50% of the memory used by GRAM goes to the perl processes.  Even on a busy CE, a lot of these have been completely idle for quite a bit of time (I think).  Can they be killed after a given idle timeout?  We could really use the memory savings.

Original reported to me from OSG via email

The issue is reported in their tracker: https://ticket.grid.iu.edu/goc/12064

We've noticed that a few job-managers will get into a state where they believe they are shutting down, but never shutdown.  The symptom is that all jobs for that user fail and we get this message repeatedly in the log:

ts=2012-03-08T10:10:45.928806Z id=24514 event=gram.signal.end level=WARN gramid=/16217969582761169661/5881249661246812557/ signal="7" jmstate=GLOBUS_GRAM_JOB_MANAGER_STATE_STOP msg="Invalid query" status=-94 reason="the jobmanager does not accept any new requests (shutting down)"
ts=2012-03-08T10:10:45.928901Z id=24514 event=gram.query.end level=ERROR gramid=/16217969582761169661/5881249661246812557/ uri="/16217969582761169661/5881249661246812557/" msg="Error processing query" status=-94 reason="the jobmanager does not accept any new requests (shutting down)

This will go on indefinitely until a admin intervenes and manually kills the g-j-m.  Jobs from other users will continue to work: it appears this is specific to a jobmanager.

As this appears to hit the most active job-managers, it is blocking these CEs from doing useful work when it happens.

Some filesystems struggle with the number of files that GRAM can accumulate in the globus_job_state directory.  One great improvement was the elimination of the per-job lockfile: this reduces the number of files by a factory of 2.

A second improvement would be to create new jobs in /var/lib/globus/gram_job_state/$LOGNAME/.

I had a ticket open for this in the OSG (https://jira.opensciencegrid.org/browse/SOFTWARE-356), but it really belongs in the Globus JIRA.

OSG would like to completely decouple GRAM from our user's home directories. A few use cases:

1) In some cases, it's a pain to automate the creation of the directories if they aren't otherwise going to exist on the CE.
2) Home directories may be on an (expensive) sitewide shared file system, and any reduction in load is greatly appreciated by the local sysadmins. GRAM files could be separately exported directly from the CE if desired, for example.
3) If local users are mapped to their local username, they'll notice lots of gram_scratch directories and unexpected files in ~/.globus that count against their quota.
4) If the Globus-related files are in a dedicated location, it's much easier to do migrate all the state to a separate machine/VM and failover to that one.

This decoupling is fairly easy to achieve.  Patches to follow.

The globus-gatekeeper fails silently for a few reasons. I think it should report errors to the user instead. We had a user hit one of these (missing certificate), and he would have solve it more quickly if an error message was printed.

The attached patch shows a tested modification to the init script.

I just noticed that we still have ws-gram on these pages:
        http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/developer/scheduler-tutorial.html

We need to review and update the documentation here.
  - Remove and WS GRAM details.
  - Remove the resource publishing section.
  - Add an overview section that describes in more details the issues with interfacing GRAM with an LRM.
  - Add a requirements section?  e.g. a host where the LRM commands and log file (for SEG) is available.
  - Change to use terms consistently.  e.g. LRM instead of scheduler.
  - should be more clear the steps required to complete the adapter.
     + step 1, step 2, ...

One of our sysadmins did me a favor and started looking at the persistent memory leaks in globus-job-manager (folks have reported 5GB g-j-m processes after several weeks).

In accept_delegation.c, the following is called:

            local_result = globus_gsi_cred_get_cert(
                        context->peer_cred_handle->cred_handle,
                        &peer_cert);

Per the globus_gsi_cred_get_cert documentation, the caller is responsible for calling X509_free on peer_cert on success.  This is not done and the cert is leaked.

Leak was added by this commit: http://viewcvs.globus.org/viewcvs.cgi/gsi/gssapi/source/library/accept_delegation.c?r1=1.37&r2=1.38 (I think)

Some XSEDE systems use SLES11, so we should add that OS to the build and test system.

Due to a user complaint, I found a case where a user's job remained in "active" state indefinitely - even hours after the job actually finished.

Using strace, I still see the condor XML file being polled and read. I suspect there is some unforeseen race condition - or parsing issue - in the "fake" SEG for Condor.

open("/var/lib/globus/gram_job_state/condor.16217795921895230381.17186126506088338356", O_RDONLY) = 14
fstat(14, {st_mode=S_IFREG|0644, st_size=8265, ...}) = 0
getuid()                                = 30177
close(14)                               = 0

I've attached the Condor log and the job state file.

In many cases with striped server setups, the majority of transfers are non-striped.  This causes less efficient transfers for those users and more resource usage for the servers.  A hybrid mode where the server runs as a standalone server until a striped command is used would enable the best of both worlds, and with simpler setup for admins.

Gayane was test threaded+encryption, and ran into an error where >1 streams data channel would sometimes fail to connect.   We traced the error to a race condition in loading the crls into the global hash table.  I believe if loading the crls is slow enough, the 2nd connection attempts to load them before the first attempt has finished (I think they should only be loaded once), which causes a failure when openssl thinks a duplicate crl was found.

possibly related to GRIDFTP-238.  multiple data channels with threaded servers occasionally fail with:
500-OpenSSL Error: a_verify.c:168: in library: asn1 encoding routines, function ASN1_item_verify: EVP lib
500-OpenSSL Error: rsa_eay.c:676: in library: rsa routines, function RSA_EAY_PUBLIC_DECRYPT: padding check failed
500-OpenSSL Error: rsa_pk1.c:100: in library: rsa routines, function RSA_padding_check_PKCS1_type_1: block type is not 01

I suspect with the proper fix for 238 this may be fixed as well.

GridFTP server creates the "/var/log/gridftp.log" file with the following permission rights: 0666, which means that the file is writable by anyone.

See below some extract from the results I got using strace on the globus-gridfpt-server execution.

open("/var/log/gridftp.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=164975, ...}) = 0

In the GridFTP configuration page, in particular in the UDT section (http://www.globus.org/toolkit/docs/5.2/5.2.0/gridftp/admin/#gridftp-config-udt), the GT website states that the administrator should use the threaded version of the server and/or the client component.
In GT 5.2 this difference has disappeared. The webpage should be updated accordingly.

Test in the existing Windows GC framework

Go through list of site commands and create synonymous top-level commands for 5.2.x where appropriate.  Work with the full protocol command list Raj is putting together.

OSCARS provides API to create virtual circuits (with guaranteed bandwidth) between a pair of endpoints. Prototype the ability to reserve bandwidth in GUC using the OSCARS API. Eventually, we would like to add this ability to GO. Part of this exercise involves providing inputs to ESnet folks on the OSCARS API (what kind of improvements are needed to their API, any additional functionality etc).

This has been requested for some time, but the new HPSS dsi impl from ncsa would require this in order to install on to a standard server.  Additionally I need to look at the ability for the dsi to return custom tags in mlsx responses.

ipc options to specify credentials to connect with, and DNs to allow connections from will provide an alternative way to connect than client delegated credentials.  future work of adding extensions to credentails may tie into this.

The debian wheezy release is scheduled for freeze in June. We should add support for this in its current state so that we can shake out any GT-related bugs prior to its release.

We currently do not create binaries for ubuntu LTS releases which are still supported by ubuntu. Create new AMIs to build and test on those, and add 5.2.x build plans.

The bamboo tasks for the 5.2.x releases create an apt or yum repo package for the packages on www.globus.org/ftppub/gt5/5.2/5.2.x/. We should add other repo packages for www.globus.org/ftppub/gt5/5.2/testing and www.globus.org/ftppub/gt5/5.2/stable, and for bamboo.globus.org artifacts for testing.

The base image used for Globus graph/nexus will be moving to ubuntu 12.04LTS at the end of April.  Add support for GT packages for that system.

Milestone for tracking 5.2.1 bugfixes in the toolkit

Some of the paths used by init scripts for gridftp and gram rely on ${localstatedir}/lock and ${localstatedir}/run to be created before running, but they aren't in the installer prefix. Either the installer should create those, or we should add some install hooks when the init scripts are built.

Update the various Common and Security frags to describe new functionality, bug fixes, changes, etc.

Our MCS hosted Bamboo server is not functioning well.  It isn't handling the numerous simultaneous running builds that we're doing for GT.  The task here is to investigate if using a jenkins integration testing server is a viable and better option.

The Graph team is already maintaining one for their needs that we can use as well.
   https://jenkins.utils.globuscs.info/

I'm getting errors like System error in mkdir: File exists on mkdir and 500-globus_xio: System error in open: Is a directory on STOR.   Neither case is giving me something close to the error I want, like 'path too long'.

I'm using files like: python print "/home/karlito/" + (("a" * 100 + "/") * 44) + "somefilename"

The usage stats server sometimes encounters a badly formatted gram packet with a missing or malformed uuid. After it fails to insert this packet, it tends to put an hour's worth of usage stats in the failed state for later. It's fairly easy to detect this type of bad packet (the uuid is missing or not in uuid form) so we should be able to discard these by filtering them out when the insert statement is generated for that packet.

The gram5_rsl_attribute_groups.attributes column, which contains a comma-separated list of rsl attributes in a job is not wide enough to handle all of the RSL attributes. I don't think this table is used in any of our queries, so it might be ok to ignore it altogether. I started a query to double the size of that column yesterday morning, but it's been running for 24 hours without any success. I'll try to see if we have any use for that table, and if not, stop trying to insert into it.

The path to the job dir created by globus-personal-gatekeeper is based on $TMPDIR, which is, on MacOS, a rather large path unique to each user. Adding the various other components to the pathname, including a redundant $LOGNAME creates a path which is longer than is valid for sockaddr_un addresses, so the job manager can't run correctly. Since the job directory gets the username and hostname appended to it, the path passed to gram can be shorted without worrying about having duplicate path names on different machines.

In globus_l_gram_file_cleanup(), the job manager allocates a string for the condor log path and doesn't free it

==26696== 10,580 bytes in 115 blocks are definitely lost in loss record 665 of 670
==26696==    at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==26696==    by 0x7B9FD15: globus_common_v_create_string (in /usr/lib64/libglobus_common.so.0.14.6)
==26696==    by 0x7B9FDCC: globus_common_create_string (in /usr/lib64/libglobus_common.so.0.14.6)
==26696==    by 0x4267BB: globus_l_gram_file_cleanup (globus_gram_job_manager_state.c:2226)
==26696==    by 0x424AE6: globus_l_gram_job_manager_state_machine (globus_gram_job_manager_state.c:782)
==26696==    by 0x423AB5: globus_gram_job_manager_state_machine_callback (globus_gram_job_manager_state.c:137)
==26696==    by 0x7B92033: globus_callback_space_poll_nothreads (in /usr/lib64/libglobus_common.so.0.14.6)
==26696==    by 0x7BB2E1F: ??? (in /usr/lib64/libglobus_common.so.0.14.6)
==26696==    by 0x409A3D: main (main.c:634)

There appears to be a couple of leaks in the stdio update processing, both in rsl merging process and the stream list replacement.

The following stacks appear multiple times in the valgrind traces.

==9995== 35 bytes in 5 blocks are indirectly lost in loss record 244 of 526
==9995==    at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==9995==    by 0x6F5B07F: globus_rsl_copy_recursive (in /usr/lib64/libglobus_rsl.so.2.7.1)
==9995==    by 0x41A46D: globus_gram_job_manager_rsl_merge (globus_gram_job_manager_rsl.c:77)
==9995==    by 0x4152F1: globus_l_gram_stdio_update_signal (globus_gram_job_manager_query.c:2067)
==9995==    by 0x4136B3: globus_l_gram_job_manager_signal (globus_gram_job_manager_query.c:881)

==9995== 120 bytes in 5 blocks are indirectly lost in loss record 341 of 526
==9995==    at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==9995==    by 0x6F59C4D: globus_rsl_value_make_literal (in /usr/lib64/libglobus_rsl.so.2.7.1)
==9995==    by 0x6F5AF1C: globus_rsl_value_copy_recursive (in /usr/lib64/libglobus_rsl.so.2.7.1)
==9995==    by 0x4233BB: globus_l_gram_job_manager_staging_add_pair (globus_gram_job_manager_staging.c:896)
==9995==    by 0x422532: globus_l_staging_replace_one_stream (globus_gram_job_manager_staging.c:308)
==9995==    by 0x4226A6: globus_l_staging_replace_stream (globus_gram_job_manager_staging.c:408)
==9995==    by 0x422243: globus_gram_job_manager_streaming_list_replace (globus_gram_job_manager_staging.c:162)
==9995==    by 0x418374: globus_i_gram_request_stdio_update (globus_gram_job_manager_request.c:2359)


==9995== 300 bytes in 5 blocks are indirectly lost in loss record 400 of 526
==9995==    at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==9995==    by 0x6F5AF06: globus_rsl_value_copy_recursive (in /usr/lib64/libglobus_rsl.so.2.7.1)
==9995==    by 0x4233CF: globus_l_gram_job_manager_staging_add_pair (globus_gram_job_manager_staging.c:897)
==9995==    by 0x422532: globus_l_staging_replace_one_stream (globus_gram_job_manager_staging.c:308)
==9995==    by 0x4226A6: globus_l_staging_replace_stream (globus_gram_job_manager_staging.c:408)
==9995==    by 0x422243: globus_gram_job_manager_streaming_list_replace (globus_gram_job_manager_staging.c:162)
==9995==    by 0x418374: globus_i_gram_request_stdio_update (globus_gram_job_manager_request.c:2359)
==9995==    by 0x41538E: globus_l_gram_stdio_update_signal (globus_gram_job_manager_query.c:2091)


==9995== 2,100 (120 direct, 1,980 indirect) bytes in 5 blocks are definitely lost in loss record 517 of 526
==9995==    at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==9995==    by 0x7BA1C99: globus_list_insert (in /usr/lib64/libglobus_common.so.0.14.6)
==9995==    by 0x423672: globus_l_gram_job_manager_staging_add_pair (globus_gram_job_manager_staging.c:998)
==9995==    by 0x422532: globus_l_staging_replace_one_stream (globus_gram_job_manager_staging.c:308)
==9995==    by 0x4226A6: globus_l_staging_replace_stream (globus_gram_job_manager_staging.c:408)
==9995==    by 0x422243: globus_gram_job_manager_streaming_list_replace (globus_gram_job_manager_staging.c:162)
==9995==    by 0x418374: globus_i_gram_request_stdio_update (globus_gram_job_manager_request.c:2359)

The gsi sysconfig library generates some error objects (returned as globus_result_t) from its internal functions, and in some cases discards those before returning a GLOBUS_SUCCESS to the caller. These fill some space unnecessarily in the result cache and waste memory.

Like GT-188, not a true leak, but (at least on linux) the regexec() call to parse the condor log will add nodes to its internal state table as it parses different inputs. These will be never be freed until a call to regfree() on the compiled regular expression.

I've seen a few valgrind stacks like this after running the client test suite against a job manager running with valgrind. I'm not sure what sort of jobs are triggering this leak, but it seems to be recurring.

==1514== 855 bytes in 15 blocks are definitely lost in loss record 365 of 376
==1514==    at 0x4C26FDE: malloc (vg_replace_malloc.c:236)
==1514==    by 0x9044B41: strdup (in /lib64/libc-2.12.so)
==1514==    by 0x50385CE: globus_l_gram_protocol_get_string_attribute (globus_gram_protocol_pack.c:2420)
==1514==    by 0x5038B86: globus_gram_protocol_unpack_job_request (globus_gram_protocol_pack.c:253)
==1514==    by 0x425B18: globus_gram_job_manager_read_request (globus_gram_job_manager_state.c:1533)
==1514==    by 0x416553: globus_gram_job_manager_request_load (globus_gram_job_manager_request.c:982)
==1514==    by 0x42EABD: globus_l_gram_startup_socket_callback (startup_socket.c:1680)
==1514==    by 0x6D032A7: globus_l_xio_read_write_callback_kickout (globus_xio_handle.c:1216)
==1514==    by 0x6D0383F: globus_i_xio_read_write_callback (globus_xio_handle.c:1184)
==1514==    by 0x6D0B38B: globus_l_xio_driver_op_read_kickout (globus_xio_driver.c:637)
==1514==    by 0x6D1678C: globus_xio_driver_finished_read (globus_xio_pass.c:1238)
==1514==    by 0x6D25B90: globus_l_xio_file_system_read_cb (globus_xio_file_driver.c:682)

There have been a large number of job failing for reasons other than user cancel. Investigate those failures, and try to find interesting relationships between the errors and service deployments or client-requested features, or LRMs used.

See https://ticket.grid.iu.edu/goc/12125

Reported by Florida shortly after upgrade to 13.35.  Backtrace:

(gdb) bt
#0  0x00000000004092e9 in globus_gram_job_manager_request_log (
request=0x7fff1bdd87d0, level=GLOBUS_GRAM_JOB_MANAGER_LOG_TRACE,
format=0x429280 "event=gram.state_file.read.end level=TRACE path=%s status=%d \n") at globus_gram_job_manager_request.c:1515
#1  0x0000000000411b03 in globus_gram_job_manager_state_file_read (
request=0x7fff1bdd87d0) at globus_gram_job_manager_state_file.c:766
#2  0x0000000000408139 in main (argc=5, argv=0x7fff1bdd9048)
at globus_gram_streamer.c:150

It appears request->config is NULL and the log function doesn't check for this.

just OPENSSL_CFLAGS and OPENSSL_LIBS should do it
1:18
OPENSSL_CFLAGS="-I ${openssl}/include" OPENSSL_LIBS="-L${openssl}/lib -lssl -lcrypto"

please put that in the INSTALL or somewhere on the web site for those of us on macs without pkg-config nor the inclination to build it and its crap dependencies.

I saw the following assertion failure, probably when the job was shutting down.

(gdb) bt
#0  0x00000034ae630265 in raise () from /lib64/libc.so.6
#1  0x00000034ae631d10 in abort () from /lib64/libc.so.6
#2  0x00000034ae6296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x000000000040cc9d in globus_gram_job_manager_stop_all_jobs (manager=0x7fff6b999180) at globus_gram_job_manager.c:2104
#4  0x00000034b361850e in globus_callback_space_poll_nothreads () from /usr/lib64/libglobus_common.so.0
#5  0x00000034b363846f in ?? () from /usr/lib64/libglobus_common.so.0
#6  0x0000000000409741 in main (argc=, argv=) at main.c:641
(gdb) up 3
#3  0x000000000040cc9d in globus_gram_job_manager_stop_all_jobs (manager=0x7fff6b999180) at globus_gram_job_manager.c:2104
2104            assert(rc == GLOBUS_SUCCESS);
(gdb) p ref
$2 = 
(gdb) p ref->key
Cannot access memory at address 0x0

This is with the latest release (13.35-0.4), *without* the OSG patch to globus_gram_job_manager_stop_all_jobs.

We're using GridFTP from GT 5.2.1 and we (Doug Strain and Neha Sharma) found an interesting bug. Normally, GridFTP maps me to the user that I am mapped to in the grid-mapfile. For instance, when I'm mapped like this:

"/DC=org/DC=doegrids/OU=People/CN=Alain Roy 424511" alainroy

I'm mapped to the alainroy user. I can easily tell which user it is with UberFTP, though the client is irrelevant:

% uberftp fermicloud084
220 fermicloud084.fnal.gov GridFTP Server 6.5 (gcc64, 1323378368-83) [unknown] ready.
230 User alainroy logged in.
UberFTP> pwd
/cloud/login/alainroy

However, if I'm mapped to a user that doesn't exist, GridFTP appears to pick the last user in /etc/passwd. For example, when alainroy is misspelled:

"/DC=org/DC=doegrids/OU=People/CN=Alain Roy 424511" alainroyy

I'm mapped to the tomcat user:

% uberftp fermicloud084
220 fermicloud084.fnal.gov GridFTP Server 6.5 (gcc64, 1323378368-83) [unknown] ready.
230 User alainroyy logged in.
UberFTP> pwd
/usr/share/tomcat5

apparently because Tomcat is the last user in the passwd file:

% tail -1 /etc/passwd
tomcat:x:91:91:Tomcat:/usr/share/tomcat5:/bin/sh

Another example:

% globus-url-copy file:///cloud/login/alainroy/shar.pl gsiftp://fermicloud084.fnal.gov/tmp/shar.pl
% ls -l /tmp/shar.pl
-rw-r--r-- 1 tomcat tomcat 55051 May 17 12:11 /tmp/shar.pl

I would think that if the user doesn't exist, something safer would happen. Probably you should deny access.

Lest this seem like a rare condition, it's pretty common for people in OSG to mistakenly authorize users that don't have accounts. People authorize whole VOs because they authorize "everyone in OSG" but regularly forget to make any of the accounts for them. So this may well be a common problem and could cause security breaches. Definitely something to fix.

If you provide us with a patch, we can ship a patched version to OSG in advance of a new release from you.

Thanks!
-alain

In trying to create update packages for some bug fixes, I noticed that our repos are missing debian source packages altogether.

The debuginfo package are missing for the RPMs for CentOS 5 and the RHEL builds. These packages contain the debugging symbols and source files so that users can use the globus tools in a debugger.

globusrun -status will crash if it fails to authenticate with the job manager process, as it tries to look into an uninitialized extensions hashtable.

The gram audit code uses globus_libc_getpwuid_r() but doesn't check that res is non-NULL, so if the current uid can't be resolved, it will record the last uid that getpwuid returns.

APS was having strange issues on a SUSE box, including firewall-like hangs with no firewall present, and slowing transfer speeds when it did work.   The machine is on a restricted network, so I tried debugging via email, unsuccessfully.  I then loaded the same distro on a vm image and tested with that, but didn't see a problem.

Suresh is going to see if he can arrange remote access to the machine, and if so I expect I could spend a day debugging directly.

XSEDE sites want to be able to install a testing version of GridFTP (or other software) in an alternate install location.  We need to figure out an acceptable way for them to do this.  It seems that building the Globus Toolkit RPMs as relocatable is probably easiest for both our team and the RPs, but we need to try it out to confirm that this strategy is viable.

Flesh out the outline for XSEDE GridFTP deployment into full documentation

Track XSEDE requirements with regard to GridFTP deployment, to inform the documentation and software release process.

From Troy Baer from XSEDE ticket 215602:

We've had a long-standing problem on Kraken where the GRAM5 service wouldn't start up properly when the system boots.  Yesterday, we discovered that this was because the globus-gatekeeper process will exit silently (i.e. without logging *anything*)
if it detects the existence of /etc/nologin.  This is highly suboptimal, and I'd like to get it fixed if possible.

(This is with gram5-5.0.4, BTW.)

Troy's recommendation:

in the presence of /etc/nologin the gatekeeper should be just like its behavior in the presence of
$GLOBUS_HOME/{etc,var}/globus-nologin -- exit, but log why you're doing.

CentOS 4 is used by XSEDE for ranger. It has an older version of many packages, so some of the build and runtime dependencies will need updates to work with it. Also, the version of yum on CentOS 4 is old and doesn't handle html redirects like the new versions, so the build-rpms script will need some tweaks to generate repos to use. This build doesn't currently need myproxy.

With native packaging, it's becoming much easier for us to push bug fixes to sites via the updates repository. Doing so, however introduces some confusion about what things like Toolkit Version 5.2.1 really mean, as doing a yum or apt install will get 5.2.1 + bugfixes and eventually fold into 5.2.2 and whatever else follows. This is somewhat complicated by situations (solaris, macos) where we don't have native packages or updaters in place. In those cases, we'd need to push out update source packages as before.

So, we need to define some behaviors related to the new updating release:
- What the 5.2 release versions mean?
  - What do you get when you install the repo packages linked to from the toolkit downloads page?
- Document how to upgrade and downgrade packages
- How we will keep documentation up to date, while still keeping relevant documentation to previous versions available
- How will we push updates to the non-native packaged world.

The globus.org website is going to become more globusonline.org and less globus.org/toolkit, so we'll need to figure out how to manage the documentation on the new site. This is somewhat related to GT-207, as it's not entirely clear what the 5.2 documentation will look like with the auto-updating release stream. At the least, the documentation parts of the http://confluence.globus.org/display/GT/GT+5.2+release+process should be updated to run on the new (virtual?) server, with perhaps some overhaul of that process to make it more automated if still needed.

The job manager crashed during a couple of tests on CentOS 4. It looks like the query callback is trying to log with a null request and this is hitting a crash. Perhaps related to the GRAM-207 and  GT-192 fixes.

The grid-mapfile-check-consistency script flags having multiple lines with the same DN as an error, but doesn't notice invalid local users except in the first line with a particular dn. I ran in to this before suggesting we indicate this tool might help detect vulnerability to GT-195

On the download webpage, add a sentence or 2 how update packages / advisories augment the 5.2.x releases.  Add a link to advisories.

http://www.globus.org/toolkit/downloads/latest-stable/

The packages globus-gram-audit globus-gridmap-callout-error exist in the installer and rpm packages, but not in our debian builds.

The CentOS 5 tests on bamboo are haning/failing because of SGE misconfigurations. It looks like the gridengine_init script is not being run---not sure if there are leftovers in the /usr/share/gridengine  directory. Also, the gridengine tools have -ge appended to them in that install (qstat-ge). The non-ge versions may be torque or gridengine depending on the order of installation.

The unique IDs of jobs that get stuck into the pending_restarts list seems to leak when a job manager is started. Also, the state file reader can leak pieces of the request structure if the request is discarded because of an LRM or service tag mismatch.

Nick B. at nersc reported a bunch of errors from a particular user/GO transfer.  The gridftp processes were hung after the errors.

The toolkit downloads pages (such as http://www.globus.org/toolkit/downloads/5.2.1/ ) contain repos and the source installer, but the repos might contain fixes not in the source installer. Add support for parsing the advisories.txt file and generating something similar to the advisories page, but only with updates relevant to that particular release.

The tasks in http://confluence.globus.org/display/GT/GT+5.2+release+process#GT5.2releaseprocess-Doccoordinatortasks should be automated so that making a release has fewer steps.
- Investigate CVS branching or automating the copy of the files
- Fix the make files so the olink and html toplevel make rules work instead of relying on find
- Use the frag generator script to create change summaries, fixed bugs, and known problems frags

The various repositories testing, stable, versioned, should be documented on the admin guide, including what they can contain and what their paths are.

Modify the admin guide or toolkit release notes to contain information about how to go about installing an update package from source or from one of our repositories. Add a link to the rss page for update announcements http://www.globus.org/toolkit/rss/advisories/5.2.rss

After the bamboo migration is complete, add triggers to automatically run the build and test tasks when the packaging metadata is modified. When these tasks complete, have artifacts published somewhere so that a periodic job on login.globus.org can download them and publish them to the testing repo.

Eric investigated relocatable RPMS in GT-202. Support for this is apparently easy to add, so we should modify our spec files to include this feature. To do so, we will need to add relocations for /usr /etc /var to each spec file.

xinetd allows you to configure a limit on connections from a particular ip address, but there is no way to limit the number of connections for the mapped user.  With the current server, it would require state files to keep track of the number of connections across processes, perhaps one per username populated with a single number.  The file would be locked and the number incremented or decremented for each connection or disconnection.   I believe the main difficulty would be ensuring that all disconnections get counted.

There are some problems with the job state files that have some effects on the scalability and performance of GRAM. For example, each file must be read and write completely in order to be able to processed, as there is no way to do random access to the file because of its variable length fields. Whenever GRAM needs to do processing on behalf of a job, it usually ends up reading the whole file, even though most of the fields are only needed when the job is submitted. Some things like stdio update and proxy refresh (which condor uses) could be handled without necessarily having to read the whole state file into memory if there was a way to do random access to get to certain pieces of the state. Also, the restart code ends up loading each state file in turn, and then doing some processing to get the job request structure into a consistent state to know what needs to be done with it.

One possibility would be to replace the state file with an sqlite databse. This would add random access to the fields of the state file (add a row to register a new client callback contact, replace a row to do stdio update, query job state ) and also allow the job manager to pull out the ids of jobs that need to poll or are ready to send a state callback without having a large set of job requests in memory at any given time.

Several OSG sites have complained about a severe decrease in scalability between GRAM2 and GRAM5.  I think this is due to the fact that we previously had one grid_monitor per DN per submit host, and now have one globus-job-manager per DN.

Adding three submit hosts does not gain a factor of 3 in scalability: everything goes through the same g-j-m process.

Even worse - a significant issue at one submit host can render the other two inoperable (by hitting the concurrency limit, or making globus-url-copy stageout stall).

Can we include the submit host in the hash calculation for the g-j-m tag?

Our large PBS sites have reported issues with GRAM5 losing track of jobs when SEG is enabled.

If the SEG misses an event, the job might indefinitely stay in Idle or Running state.  After a few weeks of running, if there's enough Idle jobs, the various pilot factories will stop submitting - possibly causing the site to become non-operational.

The only workaround I have found is to restart the g-j-m without the SEG, let it do the explicit qstat, and then restart it again with the SEG.  This is a complex procedure for sites to follow.

Is it possible to have a "hybrid SEG" mode, where the job states are explicitly queried once every 1-4 hours?

UChicago IdP releases the eduPersonPrincipalName and eduPersonTargetedID attributes. https://test.cilogon.org/ can process the attrobutes and issues user credentials with the reflecting extensions (ePPN 1.3.6.1.4.1.5923.1.1.1.6, ePTID 1.3.6.1.4.1.5923.1.1.1.10). The ePTID can be then mapped by a GridFTP authz callout to a unix account. A request for such a callout has been submitted as the GO ticket https://support.globusonline.org/tickets/300237.

Trying to understand some old gram code, I ran into the use of the priority queue and got confused because of lack of documentation.  Add doxygen comments for the globus_priority_q structure and API to make it easier to understand what's going on.

after building gridftp natively for windows, I realized that gsissh's dependence on cygwin still meant that we needed the entire set of globus cygwin libraries.  http://www.nomachine.com/contributions  has a patched openssh for windows, and I should be able to apply Jim's gsi patches to that to get a native gsissh client.

Partially port and build gridftp server for windows.  Ignore the process handling such as fork and chroot, ignore user and file handling issues such as setuid/getuid and *pwent/*grent.  The server will run only as a single process (should be inetd-able), and only with permissions of the user running the process.  File listings and stats will have no ownership or permissions info.  Transfers, security, sync, etc, should all be fully functional, and it should do everything needed for the LTA.

discovered this while testing in prep for 5.2.2.

On this page (for 5.2.1) http://globus.org/toolkit/downloads/latest-stable/
The link for "Ubuntu 12.04 LTS (testing)" returns
"The requested URL /ftppub/gt5/5.2/5.2.1/installers/repo/globus-repository-5.2-stable-precise.0.3_all.deb was not found on this server."

The correct URL should be this:
http://www.globus.org/ftppub/gt5/5.2/5.2.1/installers/repo/globus-repository-5.2-stable-precise_0.0.3_all.deb

Looks like there is a missing -->  "_0"

The GSSAPI specs mandates that the mech_type and src_name parameters of gss_accept_sec_context() are filled upon the successful completion of a context establishment sequence. The Globus implementation returns these values only in the middle of the process, which makes it difficult to use it with stacked gssapi pseudomechanisms and glues. The attached patch addresses this shortcoming. Please note that even if it's not required, it'd be very useful to have the mech oid returned by every single call of the function (not just the last one) as indicated by the patch.

When logrotating was added to the gatekeeper, we didn't take into account the fact that the various processes do not re-open (or cannot re-open) the file descriptor for the gatekeeper log.

This means that the globus-job-manager processes in particular cannot reliably log to that file: after a day of running, they log to a deleted file descriptor!

This causes us to lose quite a few interesting error messages.

When looking for the CA dir in globus_gsi_system_config.c, if a home directory is not found, the logic will pop off the error object and proceed to check elsewhere.

However, it appears there are two error objects in the stack (one for the missing home dir, one for the missing directory).  So, we still get

"File does not exist: /grid_home/uscmsPool014/.globus/certificates is not a valid directory"

in the logs (well, in strace).  Due to GT-233, it's a bit hard for me to follow whether this results in the entire operation failing (as the result has errors associated with it).

As far as I can tell, the globus-job-manager does not reload CRLs while it is running.  This means that, eventually, the CRL loaded into the process will expire and all SSL operations will be denied.

I've not had the time to reproduce, but I did find some messages along the lines of "CRL expired" in a non-functioning globus-job-manager process.  These cleared up when I killed off the process.  I also found an OpenSSL bug report stating that these do not get refreshed.

Obviously, we do not want to reload the CRL for every request - but probably cache it for some reasonable amount of time.

Does CRL verification even make sense within the g-j-m process?  If not, maybe we can avoid writing a lot code?

The makefile in globus-gram-audit does

mkdir -p 01733 $(DESTDIR)$(localstatedir)/lib/globus/gram-audit

instead of

mkdir -p -m 01733 $(DESTDIR)$(localstatedir)/lib/globus/gram-audit

so instead of setting mode it creates a number directory.

The page http://www.globus.org/toolkit/versioning.html should be updated to include info relating to the 5.2 release stream.

As presented in http://www.rjh.org.uk/ftp-report.html, there's no standard way for a (grid)ftp server to allow a client to set remote file times.

With the -sync capability introduced in 5.X, there is an expectation in the user community that "some level of sync will preserve timestamps" ala rsync... not an entirely unreasonable expectation.  There's also the associated "make permissions match", which at least is possible via the GridFTP protocol if not GUC directly.... While I'd love to request GUC -sync options to preserve timestamps and permissions, this improvement needs to happen first.

thanks,

--Chan

I am seeing this on Mac OS X from the 5.2.2rc1 installer:

/usr/bin/gcc -g   -m64 -fno-common  -Wall -g -m64 -fno-common -Wall  -Wall -Wpointer-arith -Wsign-compare -Wformat-security -Wno-pointer-sign -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -fno-builtin-memset -fstack-protector-all  -I. -I.. -I. -I./..  -I/include/globus -I/ -no-cpp-precomp -I/Users/jbester/development/globus_5_2_branch/5.2.2rc1/gt5.2.2rc1-all-source-installer/globus-location/include/globus -I/Users/jbester/development/globus_5_2_branch/5.2.2rc1/gt5.2.2rc1-all-source-installer/globus-location/include/globus/gcc64dbg -no-cpp-precomp -I/Users/jbester/development/globus_5_2_branch/5.2.2rc1/gt5.2.2rc1-all-source-installer/globus-location/include/globus -I/Users/jbester/development/globus_5_2_branch/5.2.2rc1/gt5.2.2rc1-all-source-installer/globus-location/include/globus/ -I/Users/jbester/development/globus_5_2_branch/5.2.2rc1/gt5.2.2rc1-all-source-installer/globus-location/include/globus  -DHAVE_CONFIG_H -c port-tun.c
port-tun.c:111:20: error: /net/if.h: Input/output error
port-tun.c: In function 'sys_tun_open':

The include lines in the openbsd-compat directory contain -I${prefix}/include/globus -I${includedir}/${GLOBUS_FLAVOR_NAME}, but those macros are not defined. On Mac OS X, /net is a special filesystem, so #include  with -I/ causes an I/O error. The top-level Makefile has those macros defined, but they aren't propagated to all of the makefiles.

The debian package builder for 5.2.2rc1 sets GLOBUS_VERSION to 5.2.1 before building globus-common, so it ends up with the wrong version string in the globus-version command.

See original description of the problem sent by Frank Scheiner:

"
Dear all,

during software evaluation in PRACE-1IP we recognized some odd behaviour of guc
when it is killed with Ctrl+C (SIGINT). You can find an elaborate background on
[1].

The problem is, that guc exits with 0 when SIGINTed.

The reason is the following:
Currently guc (<=v8.4) does exit on SIGINT "unconventionally", meaning it
catches a SIGINT, but after doing its internal cleanup and writing out a
possible dumpfile, it does *not* reset the SIGINT handler to the default SIGINT
handler and kills itself with SIGINT, but simply exits normally (leading to "0"
as exit value in the bash shell).

I would expect it to exit with "killed by signal" (128 + signal number, which is
2 for SIGINT (see [2])). I use the exit values of guc to decide if guc worked or
not, or if it was was killed by a signal.

I think the issue could be solved by integrating a clause to detect if guc was
interrupted (globus_l_globus_url_copy_ctrlc == GLOBUS_TRUE), then restore the
original SIGINT handler and at last kill itself with SIGINT.
The relevant part of the code for the clause is the end of main() in
"globus-url-copy.c" (l.2043).

I would appreciate it very much if the Globus developers could do something
about this issue, as it would help interoperability with other tools very much.

Best regards
Frank Scheiner
_____________
[1] 
[2] 

--
Frank Scheiner

High Performance Computing Center Stuttgart (HLRS)
Department Project User Management & Accounting

Email: [email protected]
Phone: ++49(0)711/685-68039
"

See original description of problem sent by Frank Scheiner:

"
Dear all,

during software evaluation in PRACE-1IP we recognized that guc does not preserve file permissions from the local/source side to the remote side.

I made some tests with two Debian 6 driven VMs using the latest GridFTP server packages (v6.11) available and a Ubuntu 11.04 driven workstation with the latest guc package (v8.4) to get some more insight to this issue.

The v8.4 guc has an option named "-preserve" that should - according to [1] - preserve the permissions of files when transferring. It's not documented in the help output, but it's in the source code (according to [1] since 2010).

For these tests I created a source directory having the following content:

"
/home/johndoe/my_special_perm_files$ ls -la
total 24
drwxr-xr-x 2 johndoe johndoe 4096 Jun 20 07:58 .
drwxr-xr-x 4 johndoe johndoe 4096 Jun 20 07:57 ..
-rw------- 1 johndoe johndoe 1024 Jun 20 07:59 0600
-rw---x--x 1 johndoe johndoe 1024 Jun 20 07:59 0611
-rw-r-x-wx 1 johndoe johndoe    0 Jun 20 07:58 0653
-rwx------ 1 johndoe johndoe 1024 Jun 20 07:59 0700
-rwxr--r-- 1 johndoe johndoe 1024 Jun 20 07:59 0744
"

As you can see, the files are named like their permission mode. One file has zero length, to see if this would have an effect. I used the following command to transfer those files from source to destination controlled from my Ubuntu 11.04 driven workstation (reformatted for better reading):

"
$ globus-url-copy \
  -cd \
  -preserve \
  gsiftp://gridftp.zeta.orion:2811/~/my_special_perm_files/* \
  gsiftp://gridftp.epsilon.terra:2811/~/my_destination_dir/
"

The resulting dir contents look like that:

"
/home/johndoe/my_destination_dir$ ls -la
total 24
drwxr-xr-x 2 johndoe johndoe 4096 Jun 20 08:14 .
drwxr-xr-x 3 johndoe johndoe 4096 Jun  8 10:22 ..
-rw-r--r-- 1 johndoe johndoe 1024 Jun 20 08:14 0600
-rw-r--r-- 1 johndoe johndoe 1024 Jun 20 08:14 0611
-rw-r--r-- 1 johndoe johndoe    0 Jun 20 08:14 0653
-rw-r--r-- 1 johndoe johndoe 1024 Jun 20 08:14 0700
-rw-r--r-- 1 johndoe johndoe 1024 Jun 20 08:14 0744
"

All files have 0644 as permission mode - the default for newly created files. Obviously the "-preserve" option does not work as expected.

I then again had a look into the source code of guc (available from [2]) and found another option that looks like it could do the job. It's called "-copy-perms" and in lines 4045 to 4048 (of "globus_url_copy.c") you can see, that it's really intended to do the same job:

"
[...]
    case arg_preserve:
    case arg_perms:
        guc_info->perms = GLOBUS_TRUE;
        break;
[...]
"

Hence I tried the above guc command with "-copy-perms" instead of "-preserve", but this yields the same result, all files 0644. I saw how the "guc_info" struct is evaluated for "delayed passive" in lines 5859 to 5863 (of "globus_url_copy.c"):

"
[...]
    if(guc_info->delayed_pasv)
    {
            globus_ftp_client_operationattr_set_delayed_pasv(
                ftp_attr, GLOBUS_TRUE);
    }
[...]
"

...but there is nothing similar for the perms variable. Hence I assume this was forgotten or is not implemented yet.

I think this "preserve permissions" functionality is something really useful and also needed. Hence the Globus developers  should be contacted to have a look into this issue.

Best regards
Frank Scheiner
___________
[1] 
[2] 
[3] 

--
Frank Scheiner

High Performance Computing Center Stuttgart (HLRS)
Department Project User Management & Accounting

Email: [email protected]
Phone: ++49(0)711/685-68039
"

The connection between the frontend and backend is duplicated needlessly each time a data channel is torn down and recreated.

The data handle can leak when using mode S.  Various small leaks in setup and config.

This can cause the build to fail.

MFMT on a windows server results in the actual file time set being offset by the timezone.

Gayane saw this during her high threading security overhead tests.  Looks like multiple markers near the end of transfer can overlap and one will double-free.

Prep for native GC win public release.

Mattias sent some compile patches for S390, Hurd, and mingw32 to satisfy debian build process. Merge those and test them.

Replace the groupinstall targets with metapackages that contain the dependencies, so that we can use the similar install methods for suse and other rpm systems.

The accompiler.m4 entry for S/390 uses -m32 but should use -m31 to generate code for the S/390 Linux ABI

The gass cache program package is missing a compile dependency on globus_gass_transfer.

The globus gatekeeper and job manager use SA_NOCLDWAIT which apparently is missing on gnu hurd. Mattias provided a compile patch to fix the compile. It's unclear whether these will work with the patch, but they will at least compile.

The gridftp server passes a dynamic string to fprintf, which can cause problems if the string contains %.

The gridmapdir code doesn't build on non-posix systems as it requires opendir. Mattias provides a patch to #ifdef out that code for win32.

We are living dangerously in GO land by doing a strstr() based check of the error text to determine if MLST failed due to file not found, as opposed to an internal server error.

When  gpt_create_automake_rules processes man pages, it creates filelist-man* targets for every man section present each time it encounters a different man section. This results in duplicate targets and warnings during builds.

For example, job manager includes man1, man5, and man8 section pages; when the Makefile.in is updated, it contains 3 copies of each of the targets filelist-man1, filelist-man5, and filelist-man8 because it processed them 3 times as it encountered each section.

GridFTP server from GT 5.2.1 with an auth callout pointed by GSI_AUTHZ_CONF:

# export GSI_AUTHZ_CONF=/usr/local/packages/globus-5.2.1/etc/gridmap_eppn_callout-gsi_authz.conf
# cat $GSI_AUTHZ_CONF
globus_mapping libglobus_gridmap_eppn_callout.so globus_gridmap_eppn_callout

Callout dll loader tries to load a callout library specified above but with a flavor string concatenated:

# globus-gridftp-server -p 2811 -d ALL -debug
[6558] Thu Aug  2 05:37:56 2012 :: GFork functionality not enabled.:
globus_gfork: GFork error: Env not set
[6558] Thu Aug  2 05:37:56 2012 :: No configuration file found.
[6558] Thu Aug  2 05:37:56 2012 :: Server started in daemon mode.
[6558] Thu Aug  2 05:38:01 2012 :: New connection from: transfer.api.globusonline.org:44513
[6558] Thu Aug  2 05:38:02 2012 :: transfer.api.globusonline.org:44513: [CLIENT]: USER :globus-mapping:
[6558] Thu Aug  2 05:38:02 2012 :: transfer.api.globusonline.org:44513: [SERVER]: 331 Password required for :globus-mapping:.
[6558] Thu Aug  2 05:38:02 2012 :: transfer.api.globusonline.org:44513: [CLIENT]: PASS dummy
[6558] Thu Aug  2 05:38:02 2012 :: transfer.api.globusonline.org:44513: [CLIENT]: PASS dummy
[6558] Thu Aug  2 05:38:02 2012 :: transfer.api.globusonline.org:44513: [SERVER]: 530-Login incorrect. : globus_gss_assist: Error invoking callout
530-globus_callout_module: Error with dynamic library: couldn't dlopen libglobus_gridmap_eppn_callout.so_gcc64dbg: file not found
530-
530 End.

This causes confusion, but nothing is accessible that shouldn't be.

~ may be mapped to / if there users home directory is not accessible, for instance if they only allow drive D.  In this case, ~ lists the available drives correctly, but ~/D does not correctly translate to D:\

The installer tries to add GPT's perl5 library path to the PERL5LIB environment variable, but in doing so mangles the old PERL5LIB value. It uses

export PERL5LIB="$(GPT_LOCATION)/lib/perl:$(shell echo $PERL5LIB)"

but $PERL5LIB needs a double $ to get things passed to the shell command like this:

export PERL5LIB="$(GPT_LOCATION)/lib/perl:$(shell echo $$PERL5LIB)"

It builds the full path correctly but fails map that back to a windows formatted path.

In globus_gsi_cred_handle.c, fucntion globus_i_gsi_cred_goodtill:

        result = globus_gsi_cert_utils_make_time(
            X509_get_notAfter(current_cert),
            &tmp_goodtill);

This assumes that the value of X509_get_notAfter is ASN1_UTCTIME, which is not always the case. It has value ASN1_TIME, which may be UTCTIME or GENERALIZEDTIME. To make matters worse, if the cert does use generalized time, no error is returned, it just mis-interprets the date.

The proper place to fix this is probably globus_gsi_cert_utils_make_time, in cert_utils. It's probably used in other locations as well.

Prior to openssl 0.9.6, it appears that openssl did use UTCTIME. From CHANGELOG in the openssl source:

Changes between 0.9.5a and 0.9.6  [24 Sep 2000]
...
  *) Various fixes to use ASN1_TIME instead of ASN1_UTCTIME.
     Also change the functions X509_cmp_current_time() and
     X509_gmtime_adj() work with an ASN1_TIME structure,
     this will enable certificates using GeneralizedTime in validity
     dates to be checked.
     [Steve Henson]

If we require compatibility with 0.9.5a and earlier, we may need some special handling.

The globus-redia program in 5.2.2 has an implicit dependency on ldl which it isn't linked with explicitly. On some systems (fedora 16), this causes a link failure. The easiest fix is probably to just use lt_dlopen and company instead of dlopen since the libtool library is linked with it.

It looks like windows is weird about trailing spaces and dots.

mkd /c/tmp/dirSP
mlst /c/tmp/dirSP
will work, but the dir being created and checked is just /c/tmp/dir

only when you try to access /c/tmp/dirSP/file will it fail.

Apparently I can force the creation by using a UNC path.

Need to add GridFTP commands to enable transfers between a GridFTP server and S3.  Additionally, we can use the same commands to use a GridFTP server to facilitate a S3 transfer over the GridFTP data channel.

Rough plans for the commands:

HTTP GET url  -- transfer url to the specified path, or to the client via the data channel.
HTTP PUT url  -- transfer to the url from the specified path or from the client via the data channel.

Partial gets and multi-part puts should be supported.

Other options including setting headers and auth tokens should be considered.

The /etc/globus/globus-[jobmanager].conf file are not listed as config files in the various globus-gram-job-manager-[jobmanager] RPMs. If people edit things (like the path to the Condor binaries, which is reasonably common), their changes are lost on update. If they would be marked as configuration files, there wouldn't be a problem.

Could this be fixed in a future release?

Thanks!

While debugging to http://ticket.grid.iu.edu/goc/viewer?id=12486 I noticed that a number of old jobs were not being cleaned up when the job manager restarts. The culprit seems to be the seg_jobmanager_module which doesn't normalize the date string before looking to see if an old file exists. That is causing the reply of the 1st log file of the month to fail because GRAM looks for the 32nd log of the previous month. When that fails, it increments again and normalizes, getting to the 2nd log of the new month.

The problem is more of a design limitation of getting the information from the backend to the frontend, so I don't see an obvious fix.  If it is fixed, the logs and usage database would contain multiple addresses for striped transfers, so any outside parsing that is done would need to handle that.

The config->extra_envvars member is being freed with free(). This is incorrect with the implementation as a list of variable names, as the list contents must be freed with free() but the list nodes with globus_list_remove or globus_list_destroy_all. This has caused crashes on debian 7 at least.

There is a gridmap callout to map myproxy ca certificates to local user accounts in 5.0 branch, but it's not natively packaged and in 5.2 branch.

The proxy keys generated by grid-proxy-init and the delegation protocol default to 512 bits. OpenSSL 1.0.1 adds support for TLS v1.2 that uses SHA-512 which fails when used with such small keys. Perhaps it's time to move that default to 1024 or higher.

$ ls go#ep1/home/tuecke (as karlito)
500 Command failed : Path not allowed.
$ ls go#ep1/home/nosuchuser
500 Command failed : Path not allowed.

But the ~syntax leaks it:
$ ls go#ep1/~tuecke
500 Command failed : Path not allowed.
$ ls go#ep1/~nosuchuser
500 Command failed : Cannot expand ~

Here are the additional tasks to get a HTTP 0.2 version

- HTTP v .2
        + multi-part PUT
        + multi-stream GET
        + Firm up additional command syntax and doc

        + http XIO Driver for GridFTP/S3
                - need a gridftp http client (GO/fxp)

Prototype a single port FTP transfer. Keep in mind ideas from Mode F - http://confluence.globus.org/display/~karl/Mode+F+Notes
but this isn't necessarily dependent on mode F.

The SEG module for PBS is sometimes missing events in the middle of the day on systems with NFS mounted LRM logs. The log parser should be made more robust so that it can handle transient errors better and not miss things that might have happened.

We provide a bunch of tools in GT, and many of them have the behavior of writing help output to standard error, so pagers aren't able to handle it. Make a pass through the executables and scripts and make them less annoying. See https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=3310 for an older report of this issue.

From https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=5580

Description From Ben Clifford

If a proxy has less than 3(?) hours validity remaining, globus-job-run issues
an error and refuses to submit a job because there is not enough time. Often
the remaining proxy lifetime is adequate; the construction of of a proxy
pseudolifetime smaller than the actual lifetime is in those situations
frustrating.

The ERROR could become a WARNING.

There is a Jenkins a REST and python api to interact with builds and artifacts. We should use one of those to create a script to download packages from jenkins and update our public repos on globus.org

From Horst:

I just looked at the new lsf.pm, and I don't see any of the mods we put
into our old osg-1.2 lsf.pm, so I'm wondering what the plans are for
this, or if we should just plan on hacking this into the new version
again once it's stable ...

I'm specifically talking about the declaration of the 'exclusive' flag,
which in turn sets "#BSUB -x\n". There is also a corresponding entry
in lsf.rvf, the "Attribute: exclusive".

At some point we were also playing around with $mpirun in lsf.pm, but
that's been a long time ago, and I don't remember if we ever actually
used that for anything.

But I think the 'exclusive' / "#BSUB -x\n" declaration should be in the
default jobmanager, since it will be used extensively in the ATLAS
AthenaMP whole node job configuration soon.

Another question I had was that the old lsf.rvf contained an
'Attribute: queue', which is no longer in the new lsf.rvf.
Does this mean that in the new osg-3 version of the LSF jobmanager,
we no longer need to declare every single LSF queue in lsf.rvf,
and they will be picked up automagically? We would be STRONGLY in favor
of that, since it was quite a pain to keep these queues updated
manually in lsf.rvf, since new LSF queues are created all the time ...

There are two scripts, rectify-spec-versions and rectify-debian-versions which do the same general thing for each of the native package metadata types. They update package version numbers to reflect gpt version numbers and then update dependencies so that they match the current gpt dependency versions, and modify the changelog to match the latest version update. These programs should be combined into a single process which has default behavior so that there isn't a need for a lot of command-line options to use them.

http://www.globus.org/toolkit/downloads/ still points to GT 5.2.1 as recommended 5.2 release.

The original report comes from Frank Scheiner (HLRS) on behalf the PRACE project. In brief using pipelining and reliability options with the GUC client do not work well together. I've added the full report in the attached file (FrankScheiner-GridFTP-GUC-report.txt).

Best wishes,

Emmanouil

Here is the note from Charles Bacon from ALCF:

On Sep 27, 2012, at 1:28 PM, Charles Bacon wrote:

There is no special handling for filenames that contain spaces, which breaks most CSV parsers.  I'm stuck using a regex to parse it, which is (I measured) about 20 times slower.  It would be nice if you ran whatever the standard "I need to enclose this value in delimeters because it has a weird character in it" that CSV parsers expect.  Jamming quotes around it is perfect, but I forget what you're supposed to do when the thing you're enclosing contains the enclosing character itself.

Just blowing off steam, feel free to file this into the bitbucket.  I'm just parsing about 2.5GB of xferlogs from ALCF, and the filename with spaces didn't show up until the 3.1millionth row or so, just about when I thought I was done.  :-)

The next Ubuntu release is scheduled for October 18. Sometime before then, we should grab the latest snapshot and prepare a build image and build/test jobs to have a build ready for the release date. After the release date, we should be able to drop support for Ubuntu 10.10 and 11.04, as they will no longer be supported by Ubuntu.

The next Fedora release is scheduled for November 27. Sometime before then, we should grab the latest snapshot and prepare a build image and build/test jobs to have a build ready for the release date. After the December 27, we can drop support for Fedora 16 as it will no longer be supported by Fedora.

On Mac OS X, the system-provided global variable 'environ' may not be directly accessible to shared libraries. This affects globus_libc_setenv.c in globus_common, which directly manipulates the environ variable. Directly accessing environ appears to work in many instances. But one instance where it does not work is when Globus is built on Mountain Lion (10.8.x) and run on Lion (10.7.x). You get a runtime error about being unable to find _environ in /usr/lib/libSystem.B.dylib.

According to the man page for environ, shared libraries that need to access environ can call _NSGetEnviron(), declared in , which returns a pointer to environ.

When transferring a file from globus-url-copy to the gass-server
library, if the destination filename cannot be opened, globus-url-copy
can end up blocking for a long period of time (15+ minutes). The gass
server has closed the connection, while globus-url-copy has the
connection open in CLOSE_WAIT state.
For this bug to trigger, I needed a moderately-sized file (400k) and a
wide distance between machines (UW and BNL). Otherwise, the transfer
fails immediately with this error:
   error: [globus_l_gass_copy_gass_write_callback]: gass_transfer_request_status: 3

Both XSEDE and OSG are investigating various ways to describe resources via extensions to RSL. OSG has a patch to add xcount (with some vague semantics) to the LSF resource manager. XSEDE is looking to adding something similar to the PBS based on older TG patches which define xcount and host_xcount. We'd like to come up with some coherent solution for both platforms. One thing to investigate is how JSDL and BES profiles implement resource constraints, so that we can do something compatible with that if we end up with a conversion of GRAM to be a JSDL-consuming service.

OSG has asked that the default log level be changed to INFO, so that they can trace job executions, but not see all of the other internals about what GRAM is doing. I think this can be done by pushing a few things from INFO to DEBUG, and perhaps a few things from DEBUG to TRACE so that the lower-verbosity log levels are more useful.

OSG is trying to use the service name and service tag to isolate the "managedfork" service (which is condor), but reports that there are some issues with those methods not completely isolating services from each other, so that condor and managed fork services end up mistakenly dealing with each other's jobs. Investigate this and come up with a solution to this problem.

There have been some intermittent errors related to ec2 nodes getting blocked by the DOE firewall, so that they cannot check out code from our cvs repository. Perhaps medium term, we should move the source repository to github or some other externally-managed system, but in the meantime, it would be helpful for our build and test runs to have a mirror of the repo on some node that is outside of the DOE firewall.

OSG has a DSI for hdfs, which we'd like to include in the GCMU native packages, so that it becomes easy to install a GridFTP server to deal with support for it. We'd like to get the package updated for native packaging with GT5.2 and add dependencies and hooks into the GCMU scripts to handle the integration with HDFS. There is a HDFS module that maps to an underlying POSIX filesystem, so we can use that to test the configuration of the gridftp server hdfs integration as part of our standard build and test process without having to configure a filesystem cluster.

The native packages of globus-scheduler-event-generator-progs fail to uninstall cleanly because they have an unexpressed dependency on globus-common-progs, which may get removed prior to the pre-uninstall script being run.

Original post: https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=1234

From Bugzilla:

globus_ftp_control_data_read() sets EOF on callbacks once all data is
delivered. It is the responsibility of the calling process to inform all
threads that EOF has been received so that no further reads are registered.
There is a short window between which the library sets the handle state to
EOF and the callback informs all threads that EOF has been received. During
this time, any new buffers introduced will result in an error.

This happens in the simplest of cases. If you register 2 reads with
globus_ftp_control_data_read() on a STREAMS mode data channel with each
buffer length = X and file size = Y where X > Y, then there is a chance that
you will receive EOF on the first buffer. The second buffer could still be
in the initial error-checking phases of globus_ftp_control_data_read()
resulting in the error.

This bug still exists in the ftp control libraries distributed with GT 5.2.2. This is a problem for client and DSI authors. It would be great if we could get a fix or work around for this race condition. Even if we had someway to distinguish EOF-related errors from other errors returned by globus_ftp_control_data_read(), it would help tremendously.

If a user's certificate expires during an ftp session and the client attempts to transfer another file, globus_ftp_control_data_query_channels()  will segfault. From the attached stack trace, it appears as though the data channel is closing. I presume this is due to DCAU failing.

OSG ran into a problem where they added a blank line to the rvf file, and it caused the service to fail to parse and thus reject all jobs.

While stress testing the GridFTP server w/ the HPSS DSI, the server became very unstable with moderate load on the system (~20 clients and servers). The server would crash in numerous locations with what appeared to be memory corruption. After debugging with valgrind and dmalloc, a common theme occurred;  somehow the globus_l_gfs_data_operation_t has already been destroyed before the call to globus_l_gfs_finished_command_kickout(). The same is happening with the request structure and globus_l_gfs_request_info_destroy(). The attached valgrind file shows this.

backtrace1 shows the server aborting on a failed globus_mutex_lock(). The contents of the globus_l_gfs_data_operation_t are included in the output. In this example, I had modified globus_l_gfs_data_operation_destroy() and globus_l_gfs_request_info_destroy() to overwrite the structure before free with '0xef' to be sure this corruption wasn't a write gone haywire.

backtrace2 and backtrace3 may be related, they may be separate issues. Both files show what appears to be aborts due to the server being in the wrong state. backtrace3 occurs quite frequently and usually with state GLOBUS_L_GSC_STATE_OPEN.

I was not able to reproduce this without the DSI loaded. This problem does not exist with GT 5.1.3 w/ the DSI loaded.

The order of FTP commands given were:

ALLO 52428800
MODE S
DCAU A
PBSZ 1181696
PROT C
TYPE I
PASV
STOR 50MB


Without GT 5.2.2, we lose the awesome custom commands and so we lose the ability to stage files.

A default setup of the gridftp-server (as taken from the Fedora/EPEL RPM packages) does not set the threading model to pthreads. The server will run single-threaded as a consequence.

There is reason to believe this setup may lead to performance problems no a server that has to deal with multiple connections. In at least one case, the gridftp server on a WMS running in Europe slowed to a crawl until the following line was added to the configuration file in /etc/gridftp.conf:

 $GLOBUS_THREAD_MODEL pthread

We were able to reproduce the performance difference on a testbed machine.

If there are no evident reasons why the server should not be run in a multi-threaded mode, then this should probably become the default. Either by shipping the server with the above line in the default configuration file, or making pthread the built-in default.

Cheers,

Dennis van Dok

XSEDE SD&I team is testing a gt 5.2.2 based installation process.  A gridftp test program has been provided, but returned errors.  Work with SD&I to figure out the root cause and update the installation guide and or test program as necessary.

Add required features as per sharing discussion.
-option to enable sharing, takes DN of sharing account
-processing of .globus_sharing file
-command to add additional path restrictions

Add support for mapping existing paths to virtual paths, likely via the path restriction interface.  Need to take care that any return paths via listings or error messages or other responses reflect the virtual path, not the real path.

This issue was originally reported to the Debian Bug Tracking System:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690652

possible bashism in ./usr/sbin/globus-gridftp-server-setup-chroot line 11 ($UID should be "$(id -ru)"):
if [ $UID -ne 0 ]; then

Begin forwarded message:

From: john alexander sanabria ordonez 
Date: October 17, 2012 8:58:30 AM CDT
To: [email protected]
Subject: [gt-user] who maintains the Globus Toolkit documentation?

Hello,

I did try to follow the instructions given in the Globus Toolkit documentation web page but I found a couple of gaps that mislead the deployment of basic grid services (simpleCA + gram5 + gridftp), e.g.
The documentation does not mention that the grid-ca-certs package should be installed
The documentation can be more specific about the grid-ca-create command must be run by the globus user
When this command is run by globus user some files (found in ~/.globus/simpleCA) must be copied to the /etc/grid-security directory
The documentation should provide a hint which describes what [YUM and OSG specific] repositories should be installed for setting up a machine where Globus services  would be provisioned (https://twiki.grid.iu.edu/bin/view/Documentation/Release3/YumRepositories)

I have to thank to Marco Mambelli and Jose Caballero for their hints that allow me to define the correct set of steps for installing those services.

Regards,

PS:  I wrote some Chef recipes which allow to deploy the aforementioned services in a physical or virtual machine

We'd like to demo this at SC 2012.

From Steve:

I think what we want to do is enhance the GridFTP control channel protocol, so that a GridFTP client can mediate the passing of UDP ip/port information between the two GridFTP servers. Assume we define a new "MODE U" that is the UDP with NAT traversal + UDT + mode E.  When in that mode, I think would basically replace the PASV and PORT commands with new ones that can pass a list of alternate ip/port pairs. I'm thinking as a short-term hack we can maybe use SPSV and SPOR for these, but instead of treating the list of ip/ports as striped servers, treat them as alternate UDP ip/ports.  So after passing MODE U to each side, the client would call SPSV to one server, which would then use STUN to discover its external ip/port, and would return both its external ip/port and its local (private) ip/port. The client would pass this list to the other server using SPOR. I don't recall if this exchange has to happen in both directions -- the second server also does STUN discovery, and pass that through the client to the first server.  If so, then perhaps you use SPSV and SPOR a second time in the opposite direction to pass this.  Or maybe its easer to just add new commands to do this (UPSV, UPOR).  In any case, then the servers use the ICE negotiation protocol to figure out with ip/ports to use to talk to each other.  For example, if they happen to be on the same private network, then they will talk directly using their private network addresses.  Otherwise they will use their external addresses.  I don't recall in what situations there might even be more than 2 ip/port pairs for each server -- perhaps if the server is multi-homed?

Once the two servers have found each other via UDP, then we should drop into UDT, and then drive MODE E over that.  I think the MODE E over UDT is already done. So I think the main missing bit is adding the STUN and ICE support to the UDP connection establishment.

The test stdio-update-after-failure-test is hanging in the current build on debian 7 and ubuntu 10.10. The job manager is not running when the test is hung.

Consider moving some GT web pages to a Globus supported community wiki in order to enable the community to help maintain and improve them.

From: Pete Eby 
Date: October 19, 2012 9:28:35 AM CDT
Subject: Re: [gt-user] who maintains the Globus Toolkit documentation?

I thinking having the install and quickstart guides in a wiki format
would be extremely helpful. The globus user community seems well
suited to help maintain this documentation. These guides are very
helpful, and allowing them to be updated might help to improve them
further.

Also, perhaps an additional community wiki page could be created for
trouble shooting common issues. There seems to be some "top 10" things
users encounter frequently, and this might help save them a
considerable amount of debug / research time.

Pete

On Fri, Oct 19, 2012 at 10:22 AM, Stuart Martin  wrote:
I like that idea a lot.  We use docbook to generate most of the toolkit's web content.  We'll have to think through if and where taking on a transition like that would make sense.  Maybe there are some GT pages that are best left to be generated via docbook and some that could be changed to become wiki pages.

For example, I'm thinking (partially based on the input here) that the install and quick start might be good candidates to be wiki pages.
       http://www.globus.org/toolkit/docs/latest-stable/admin/install/#gtadmin
       http://www.globus.org/toolkit/docs/latest-stable/admin/quickstart/#quickstart

Maybe this developer guide too?
       http://www.globus.org/toolkit/docs/5.2/5.2.2/appendices/developer/

Thoughts?

-Stu

On Oct 19, 2012, at Oct 19, 6:09 AM, Pete Eby wrote:

Has the idea of placing the documentation on a wiki where it could be
updated by community members been discussed? Perhaps by users with a
Globus Online login?

We want a 'soft chroot', for input and output paths (error messages).  Or, turning off paths in error messages may also work, because GO doesn't push recursive operations to the server and shows the path itself in errors.   Perhaps if that was an option?  (SITE echo_file_names_in_errors_for_non_recursive_comands=0 for GO?)

This is lower priority for demo than SITE RP, but I think it's still wanted - if not by SC, then soon after.

security features deserve a bit of explanation.

current text:  'Allow following symlinks that lead to restricted paths.'

perhaps add this:  "Even if off, symlinks to allowed paths are allowed".

also could change current text to:  'Follow symlinks that lead to forbidden paths.'  or 'Symlinks can bypass restricted paths'.

option could also be renamed to -rp-trust-symlinks.