SG Data Management Low Level Tools

From EGEE-see WIki

This guide is contributed by Institute of Physics Belgrade.

The gLite middleware offers variety of data management client tools for uploading/downloading files to/from the Grid and replicating data.These tools can be divided into two groups: high level and low level tools.

High level tools provide a high level interface (both command line and APIs) to the basic data management functionality, hiding the complexities of catalog and Storage Elements interaction and minimizing the risk of grid files corruption. It is highly recommended for every user to deal with data management through the LCG Data Management tools, also referred as lcg-utils.

Low level tools (edg-gridftp-*, globus-url-copy, srm-* dedicated commands) can be helpful in some particular cases, but their usage is discouraged for non expert users, since they do not ensure consistency between physical files placed on a storage and entries in the file catalogue. Usage of these tools might be dangerous.

The low level tools that can be used for interaction with GSIFTP servers on SEs are listed below. The GSIFTP protocol offers the functionalities of FTP (File Transfer Protocol), but with support of GSI (Grid Security Infrastructure). It is responsible for secure, fast and efficient file transfers to/from SEs. The user must have a valid proxy in order to use these commands.

edg-gridftp-exists TURL                        Checks the existence of a file or directory on a SE
edg-gridftp-ls TURL                            Lists a directory on a SE
edg-gridftp-mkdir TURL                         Creates a directory on a SE
edg-gridftp-rename sourceTURL destTUR          Renames a file on a SE
edg-gridftp-rm TURL                            Removes a file from a SE
edg-gridftp-rmdir TURL                         Removes a directory on a SE
globus-url-copy sourceTURL destTURL            Copies files between SEs

The edg-gridftp-* commands accept gsiftp as the only valid protocol for the TURL.

One can check file physically presence in a SE, regardless of its presence in the Grid catalogs using edg-gridftp-exists and edg-gridftp-ls commands. For these operations the protocol to access the SE (gsiftp), the host (SE), the access point, the file path to the directory or file one wishes to peruse must be known.

The file exists on a SE:

$ edg-gridftp-exists gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-04/filef14a998c-fccc-4efd-8e26-8e15d7bfcc90
$

There is no file on SE:

$ edg-gridftp-exists gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-04/fake_file
error the server sent an error response: 550 550 Requested action not taken.: No such file or directory.

When executed successfully edg-gridftp-ls displays contents of directory (directory can be empty):

$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-04
fileb73238d7-c1e6-4f54-aaac-145c7e4d3a4c
filef14a998c-fccc-4efd-8e26-8e15d7bfcc90

$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-04/fileb73238d7-c1e6-4f54-aaac-145c7e4d3a4c
/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-04/fileb73238d7-c1e6-4f54-aaac-145c7e4d3a4c

Directory creation:

$ edg-gridftp-mkdir gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test
$ edg-gridftp-exists gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test
$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test

$

Making a file copy in directory on SE:

$ globus-url-copy file:/home/neda/dpmtest.txt gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test/dpmtest.txt
$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test
dpmtest.txt
$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test/dpmtest.txt
/dpm/phy.bg.ac.yu/home/aegis/test/dpmtest.txt

Rename a file on SE:

$ edg-gridftp-rename gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test/dpmtest.txt gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test/testdpm.txt
$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test
testdpm.txt

Delete a file from SE:

$ edg-gridftp-rm gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test/testdpm.txt
$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test

$

Remove a directory from SE:

$ edg-gridftp-rmdir gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test
$ edg-gridftp-ls gsiftp://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/test
error the server sent an error response: 550 550 Requested action not taken.: No such file or directory.

error the server sent an error response: 550 550 Requested action not taken.: No such file or directory.

NOTE: The commands edg-gridftp-rename, edg-gridftp-rm, and edg-gridftp-rmdir should be used with extreme care, because they do not interact with any of the catalogs and therefore they can compromise the consistency/coherence of the information contained in the Grid.