SG Data Management File Names and LFC
From EGEE-see WIki
Contents |
This guide is contributed by Institute of Physics Belgrade.
The File Names
The Storage Element is the service which allows a user or an application to store data, in read-only form, for future retrieval. The stored files can not be changed unless physically removed and replaced. Different types of file names can be associated with files in gLite middleware.
The Grid Unique IDentifier (GUID) identifies a file uniquely:
guid:<36_bytes_unique_string> guid:cb169e1e-8b3d-4c71-870c-bb02eccaefc0
The Logical File Name (LFN) also known as User Alias, user friendly way to refer to a file, which can be used in the place of the GUID:
lfn:<any_string> lfn:/grid/aegis/neda/dpmtest
The Storage URL (SURL) or Physical File Name (PFN) identifies a replica in a SE (general form):
<sfn|srm>://<SE_hostname>/<some_string>
- sfn: prefix used for files located in SEs without a SRM interface (e.g. classic SE (obsolete))
- srm: prefix used for fro SRM-managed SEs (e.g disk pool manager)
In the case of SE without a SRM interface SURL gives the host name and and path to the location of the file, while in the case SRM-managed SEs the name the file receives may have nothing to do with its physical location, because virtual file system is used.
sfn://se.phy.bg.ac.yu/storage/aegis/generated/2007-08-15/filecaf3c8e1-dc51-4638-8ab0-092d21448337
or
srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-05/file7e1a4800-434b-48c3-bc42-6903491c147f
The Transport URL (TURL) is a valid URI with the necessary information to access file in a SE:
<protocol>://<some_string> gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/aegis/2007-08-15/file71f10a8e-fb0b-41c5-99c7-8e4c55ac78c6.4840.0
One of valid protocols must be used: GSIFTP, RFIO, gsidcap. TURLs are obtained dynamically from the SURL through the Information System or the SRM interface (for SRM managed SEs). The TURL therefore can change with time and should be considered only valid for a relatively small period of time after it has been obtained.
NOTE: There can be only one GUID and more than one LFNs, SURLs, TURLs for a file
File Catalog
The LCG File Catalog (LFC) is the service which maintains mappings between different types of file names (LFNs, GUID and SURL). The building block of LFC is a unique catalog which contains records with all related names for some file. The main key is LFN of a file. The catalog publishes its endpoint (service URL) in the Information Service so that it can be discovered by Data Management tools and other services (the WMS for example).
The user should interact with the file catalog through high level tools (lcg-util), which will ensure consistency between files in the SEs and entries in the file catalogue. The CLIs and APIs are avalaible for catalog interaction. Their usage requests LFC_HOST environment variable to be set. It holds the host name of the LFC server.
$ echo $LFC_HOST lfc.phy.bg.ac.yu
The directory structure of catalog looks like this:
/grid/<VO>/<subpaths> /grid/aegis/neda
NOTE: Users of a given VO will have read and write permissions only under the corresponding <VO> subdirectory.
The following CLI commands (Unix-like) can be used:
lfc-chmod Change access mode of a LFC file/directory lfc-chown Change owner and group of a LFC file/directory lfc-delcomment Delete the comment associated with a file/directory lfc-getacl Get file/directory access control lists lfc-ln Make a symbolic link to a file/directory lfc-ls List file/directory entries in a directory lfc-mkdir Create directory lfc-rename Rename a file/directory lfc-rm Remove a file/directory lfc-setacl Set file/directory access control lists lfc-setcomment Add/replace a comment lfc-entergrpmap Defines a new group entry in the Virtual ID table lfc-enterusrmap Defines a new user entry in Virtual ID table lfc-modifygrpmap Modifies a group entry corresponding to a given virtual gid lfc-modifyusrmap Modifies a user entry corresponding to a given virtual uid lfc-rmgrpmap Suppresses group entry corresponding to a given virtual gid or group name lfc-rmusrmap Suppresses user entry corresponding to a given virtual uid or user name.
All of them have lfc- prefix. Where the path of a file/directory is required, an absolute path can be specified (starting by /) or, otherwise, the path is prefixed by the contents of the LFC_HOME environment variable.
Following commands: lfc-entergrpmap, lfc-enterusrmap, lfc-modifygrpmap, lfc-modifyusrmap, lfc-rmgrpmap, lfc-rmusrmap require ADMIN privilege.
NOTE: The operations, these commands are performing, affect catalog and not the physical files that entries represent.
Examples of usage
In order to be able to use these commands the user should have valid proxy, which can be obtained using voms-proxy-init command. First, one can ask Information Service about available LFC catalog for certain VO:
$ lcg-infosites --vo aegis lfc lfc.phy.bg.ac.yu grid02.rcub.bg.ac.yu
After that necessary environment variables for communication with LFC catalog (for existing shell) can be set:
$ export LFC_HOST=lfc.phy.bg.ac.yu $ export LCG_CATALOG_TYPE=lfc
One can list LFC name server directory/file entries:
$ lfc-ls -l /grid/aegis/neda -rw-rw-r-- 1 105 101 106 Oct 05 14:55 dpm2007-10-05 -rw-rw-r-- 1 105 101 106 Oct 04 16:59 dpmtest -rw-rw-r-- 1 105 101 106 Oct 04 17:01 dpmtest1 -rw-rw-r-- 1 105 101 106 Aug 17 17:02 dpmtest2008-08-17-13 -rw-rw-r-- 1 105 101 106 Aug 17 17:02 dpmtest2008-08-17-14 -rw-rw-r-- 1 105 101 106 Aug 17 17:03 dpmtest2008-08-17-15 -rw-rw-r-- 1 105 101 105 Aug 15 11:40 setest.txt
Option -l (the long list) gives the file mode, the number of entries in the directory, the owner in alphabetic form if the user ID is defined in the passwd file else as a decimal number, the group in alphabetic form if the group ID is defined in the group file else as a decimal number, the file size, the last modification date and the file name.
If <VO> directory is listed, generated subdirectory can be seen. It is used when LFN is not specified during of upload and registration of some file:
$ lfc-ls -l /grid/aegis drwxrwxr-x 1 101 101 0 Jul 20 09:26 acko drwxrwxr-x 3 106 101 0 Jul 24 18:10 antun drwxrwxr-x 2 111 101 0 Sep 27 14:21 generated drwxrwxr-x 7 105 101 0 Oct 05 14:55 neda
One can use lfc-mkdir for creation a directory in the LFN namespace. Option -m specifies the mode to be used (default is 777). Here, user has all permissions, group can read and others can't do anything:
$ lfc-mkdir -m 740 /grid/aegis/neda/wiki lfc-ls -l /grid/aegis/neda/ -rw-rw-r-- 1 105 101 106 Oct 05 14:55 dpm2007-10-05 -rw-rw-r-- 1 105 101 106 Oct 04 16:59 dpmtest -rw-rw-r-- 1 105 101 106 Oct 04 17:01 dpmtest1 -rw-rw-r-- 1 105 101 106 Aug 17 17:02 dpmtest2008-08-17-13 -rw-rw-r-- 1 105 101 106 Aug 17 17:02 dpmtest2008-08-17-14 -rw-rw-r-- 1 105 101 106 Aug 17 17:03 dpmtest2008-08-17-15 -rw-rw-r-- 1 105 101 105 Aug 15 11:40 setest.txt drwxr----- 0 105 101 0 Oct 16 17:16 wiki
By using lfc-chmod mode can be changed (e.g. to allow others read permission). Symbolic modes are not supported yet:
$ lfc-chmod 744 /grid/aegis/neda/wiki $ lfc-ls -l /grid/aegis/neda/ -rw-rw-r-- 1 105 101 106 Oct 05 14:55 dpm2007-10-05 -rw-rw-r-- 1 105 101 106 Oct 04 16:59 dpmtest -rw-rw-r-- 1 105 101 106 Oct 04 17:01 dpmtest1 -rw-rw-r-- 1 105 101 106 Aug 17 17:02 dpmtest2008-08-17-13 -rw-rw-r-- 1 105 101 106 Aug 17 17:02 dpmtest2008-08-17-14 -rw-rw-r-- 1 105 101 106 Aug 17 17:03 dpmtest2008-08-17-15 -rw-rw-r-- 1 105 101 105 Aug 15 11:40 setest.txt drwxr--r-- 0 105 101 0 Oct 16 17:16 wiki
Now, a file dpmtest.txt can be uploaded to the Grid. This will be done by using lcg-util:
$ lcg-cr --vo aegis -d dpm.phy.bg.ac.yu -l lfn:/grid/aegis/neda/wiki/wiki0 file:/home/neda/dpmtest.txt guid:1d91e93c-164c-403a-b72c-3cd52393d484
--vo: specifying VO (aegis) -d: destination SE (dpm.phy.bg.ac.yu) -l: specifying LFN (lfn:/grid/aegis/neda/wiki/wiki0) result: the GUID of Grid file
The file is registrated in catalog:
$ lfc-ls -l /grid/aegis/neda/wiki -rw-rw-r-- 1 105 101 106 Oct 16 17:32 wiki0
Different manipulations can be done on existing file. For example, comment can be set and unset:
$ lfc-setcomment /grid/aegis/neda/wiki/wiki0 "for wiki purpose" $ lfc-ls --comment /grid/aegis/neda/wiki/wiki0 /grid/aegis/neda/wiki/wiki0 for wiki purpose
$ lfc-delcomment /grid/aegis/neda/wiki/wiki0 $ lfc-ls --comment /grid/aegis/neda/wiki/wiki0 /grid/aegis/neda/wiki/wiki0
Besides that, ownership and mode can be changed, a file can be renamed, a symbolic link can be created, a file can be removed, ... NOTE: Again, all these operations affects catalog and not a physical file.
Removing files and directories from the LFN namespace has two basic limitations:
- a file can be removed only if there are no SURLs associated to it. If SURLs exist, the lcg_util commands should be used instead.
List all replicas for given LFN: $ lcg-lr lfn:/grid/aegis/neda/wiki/wiki0 srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-16/file3baa83f0-9d63-4ba3-aafb-2e1bcecd6203 Remove file from LFC namespace: $ lfc-rm lfn:/grid/aegis/neda/wiki/wiki0 send2nsd: NS009 - fatal configuration error: Host unknown: lfn lfn:/grid/aegis/neda/wiki/wiki0: Host not known
The correct way: $ lcg-del -a --vo aegis lfn:/grid/aegis/neda/wiki/wiki0 #delete all replicas $ lfc-ls -l /grid/aegis/neda/wiki $ lfc-ls -l /grid/aegis/neda/wiki/wiki0 /grid/aegis/neda/wiki/wiki0: No such file or directory
- a directory can be removed (-r option) only if it is empty.
$ lfc-rm -r /grid/aegis/neda/wiki /grid/aegis/neda/wiki/wiki0: File exists /grid/aegis/neda/wiki: Directory not empty
The correct way: $ lfc-ls -l /grid/aegis/neda/wiki/ $ $ lfc-rm -r /grid/aegis/neda/wiki/ $ $ lfc-ls -l /grid/aegis/neda/wiki /grid/aegis/neda/wiki: No such file or directory
