SG Data Management Use Cases
From EGEE-see WIki
Contents |
This wiki page consist of set of possible situations that can be happened when Data Management high level tools (lcg-util) is used. It is contributed by Institute of Physics Belgrade. This set is not final and everyone is welcome to contribute with its experience to this page.
Access Control List
Every directory and file which is registered in LFC catalog has a list of permissions (access control list - ACL) which specify who is allowed to access or modify it. The permissions are:read (x), write (w) and execute (x). Combination of these permissions can be associated to these entities:
- a user - user
- a group of users - group
- any other user - other
- the maximum permissions granted to specific users or groups - mask
Permissions for multiple users and groups can be defined. If this is the case, a mask must be defined and the effective permissions are the logical AND of the user or group permissions and the mask.
Commands that manipulate with ACL are:
lfc-getacl Get file/directory access control list lfc-setacl Set file/directory access control list
A directory, and any file and directory being created under that directory, in LFC has also a default ACL, which is getted in process of creation of directory. After creation, the ACL can be freely changed. Below you can see example of creation of directory, registering of file and their default ACLs.
Example 1: Manipulation with file on Grid - same user, different VOs
User must have a valid proxy:
$ voms-proxy-init -voms aegis Enter GRID pass phrase: Your identity: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka Cannot find file or dir: /home/neda/.glite/vomses Creating temporary proxy .......................................... Done Contacting voms.phy.bg.ac.yu:15001 [/C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=host/voms.phy.bg.ac.yu] "aegis" Done Creating proxy ........................................................................... Done Your proxy is valid until Wed Oct 24 03:18:34 2007
Creating LFC directory wiki_acl:
$ lfc-mkdir /grid/aegis/neda/wiki_acl $ lfc-ls -l /grid/aegis/neda -rw-rw-r-- 1 105 101 105 Aug 15 11:40 setest.txt drwxrwxr-x 0 105 101 0 Oct 23 15:23 wiki_acl
Print the ACL of a directory wiki_acl (default settings):
$ lfc-getacl /grid/aegis/neda/wiki_acl # file: /grid/aegis/neda/wiki_acl # owner: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka # group: aegis user::rwx group::rwx #effective:rwx other::r-x default:user::rwx default:group::rwx default:other::r-x
Now, some file will be put to Grid and register it in LFC catalog:
$ lcg-cr -v --vo aegis -d dpm.phy.bg.ac.yu -l lfn:/grid/aegis/neda/wiki_acl/wiki_test file:/home/neda/dpmtest.txt
Using grid catalog type: lfc
Using grid catalog : lfc.phy.bg.ac.yu
Using LFN : /grid/aegis/neda/wiki_acl/wiki_test
Using SURL : srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4
Source URL: file:/home/neda/dpmtest.txt
File size: 106
VO name: aegis
Destination specified: dpm.phy.bg.ac.yu
Destination URL for copy: gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/aegis/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4.54540.0
# streams: 1
# set timeout to 0 seconds
Alias registered in Catalog: lfn:/grid/aegis/neda/wiki_acl/wiki_test
106 bytes 0.23 KB/sec avg 0.23 KB/sec inst
Transfer took 1030 ms
Destination URL registered in Catalog: srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4
guid:6a14d029-6a6c-422a-934b-89eea3b76550
It's default ACL can be seen below:
$ lfc-getacl /grid/aegis/neda/wiki_acl/wiki_test # file: /grid/aegis/neda/wiki_acl/wiki_test # owner: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka # group: aegis user::rw- group::rw- #effective:rw- other::r--
Previously user proxy was created for AEGIS VO. Now, user proxy will be created for the different VO (SEE-GRID), but a same user, and will be checked what can be done with previously registered file.
$ voms-proxy-init -voms seegrid Enter GRID pass phrase: Your identity: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka Cannot find file or dir: /home/neda/.glite/vomses Creating temporary proxy ............................... Done Contacting voms.irb.hr:15010 [/C=HR/O=edu/OU=irb/CN=host/voms.irb.hr] "seegrid" Done Creating proxy ........................................................ Done Your proxy is valid until Wed Oct 24 03:55:15 2007
List LFC directory /grid/aegis/neda/wiki_acl. Successful:
$ lfc-ls -l /grid/aegis/neda/wiki_acl -rw-rw-r-- 1 105 101 106 Oct 23 15:43 wiki_test
List all replicas:
$ lcg-lr lfn:/grid/aegis/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4
Create replica as SEE-GRID VO user for file put into Grid as AEGIS VO user. Successful:
$ lcg-rep -v --vo seegrid -d grid15.rcub.bg.ac.yu lfn:/grid/aegis/neda/wiki_acl/wiki_test
Using grid catalog type: lfc
Using grid catalog : lfc.phy.bg.ac.yu
Source URL: lfn:/grid/aegis/neda/wiki_acl/wiki_test
File size: 106
VO name: seegrid
Destination specified: grid15.rcub.bg.ac.yu
Source URL for copy: gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/aegis/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4.54540.0
Destination URL for copy: gsiftp://grid15.rcub.bg.ac.yu/grid15.rcub.bg.ac.yu:/diskpool01/seegrid/2007-10-23/file77174385-f531-4327-9e8e-8b40c3830f23.37472.0
# streams: 1
# set timeout to 0
0 bytes 0.00 KB/sec avg 0.00 KB/sec inst
Transfer took 2020 ms
Destination URL registered in LRC: srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/seegrid/generated/2007-10-23/file77174385-f531-4327-9e8e-8b40c3830f23
List all replicas. Successful:
$ lcg-lr lfn:/grid/aegis/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4 srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/seegrid/generated/2007-10-23/file77174385-f531-4327-9e8e-8b40c3830f23
Get the GUID for LFN. Successful:
$ lcg-lg lfn:/grid/aegis/neda/wiki_acl/wiki_test guid:6a14d029-6a6c-422a-934b-89eea3b76550
Copy file, created as a AEGIS VO user, as a SEE-GRID user:
$ lcg-cp -v --vo seegrid lfn:/grid/aegis/neda/wiki_acl/wiki_test file:/home/neda/seegrid.txt
Using grid catalog type: lfc
Using grid catalog : lfc.phy.bg.ac.yu
VO name: seegrid
Source URL: lfn:/grid/aegis/neda/wiki_acl/wiki_test
File size: 106
Source URL for copy: gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/aegis/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4.54540.0
Destination URL: file:/home/neda/seegrid.txt
# streams: 1
# set timeout to 0 (seconds)
0 bytes 0.00 KB/sec avg 0.00 KB/sec inst
Transfer took 1040 ms
[neda@ce neda]$ ll seegrid.txt
-rw-rw-r-- 1 neda neda 106 Oct 23 16:14 seegrid.txt
Delete all replicas. Successful only for the replica created as SEE-GRID VO user:
[neda@ce neda]$ lcg-del -v -a lfn:/grid/aegis/neda/wiki_acl/wiki_test VO name: (null) Using GUID : 6a14d029-6a6c-422a-934b-89eea3b76550 set timeout to 0 seconds httpg://dpm.phy.bg.ac.yu:8443/srm/managerv1: Permission denied srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4 is NOT deleted srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/seegrid/generated/2007-10-23/file77174385-f531-4327-9e8e-8b40c3830f23 is deleted srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/seegrid/generated/2007-10-23/file77174385-f531-4327-9e8e-8b40c3830f23 is unregistered
[neda@ce neda]$ lcg-lr lfn:/grid/aegis/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4
Print ACL of a file wiki_test. Successful:
[neda@ce neda]$ lfc-getacl /grid/aegis/neda/wiki_acl/wiki_test # file: /grid/aegis/neda/wiki_acl/wiki_test # owner: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka # group: aegis user::rw- group::rw- #effective:rw- other::r--
Try to modify exiting ACL (as SEE-GRID VO user; note that the file was created by same user but for different VO). No success:
[neda@ce neda]$ lfc-setacl -m g:seegrid:rwx /grid/aegis/neda/wiki_acl/wiki_test /grid/aegis/neda/wiki_acl/wiki_test: Invalid argument
Example 2: Manipulation with file on the Grid - different users, same VOs
In this example the same file as above is used. Below is shown what with this file can do another AEGIS VO user. The default ACL has not been changed, which means that group has the same permissions as user, who put file into the Grid.
$ lfc-getacl /grid/aegis/neda/wiki_acl/wiki_test # file: /grid/aegis/neda/wiki_acl/wiki_test # owner: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka # group: aegis user::rw- group::rw- #effective:rw- other::r--
For the user who belongs to same VO is possible to list replicas, create replicas, copy file from Grid and use other commands from lcg-util. You must be aware that she/he with these permissions can delete and unregister all replicas of the file, even the original one. If you do not want to allow this, you must change permissions.
$ lcg-lr lfn:/grid/aegis/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4
$ lcg-rep -v --vo aegis -d grid15.rcub.bg.ac.yu lfn:/grid/aegis/neda/wiki_acl/wiki_test
Using grid catalog type: lfc
Using grid catalog : lfc.phy.bg.ac.yu
Source URL: lfn:/grid/aegis/neda/wiki_acl/wiki_test
File size: 106
VO name: aegis
Destination specified: grid15.rcub.bg.ac.yu
Source URL for copy: gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/aegis/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4.54540.0
Destination URL for copy: gsiftp://grid15.rcub.bg.ac.yu/grid15.rcub.bg.ac.yu:/diskpool01/aegis/2007-10-23/filec7bee9b4-ba20-47a5-a86f-ce7045bcfa9e.37541.0
# streams: 1
# set timeout to 0
0 bytes 0.00 KB/sec avg 0.00 KB/sec inst
Transfer took 2020 ms
Destination URL registered in LRC: srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/filec7bee9b4-ba20-47a5-a86f-ce7045bcfa9e
$ lcg-lr lfn:/grid/aegis/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4 srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/filec7bee9b4-ba20-47a5-a86f-ce7045bcfa9e
$ lcg-lg lfn:/grid/aegis/neda/wiki_acl/wiki_test guid:6a14d029-6a6c-422a-934b-89eea3b76550
$ lcg-cp -v --vo seegrid lfn:/grid/aegis/neda/wiki_acl/wiki_test file:/home/ivana/seegrid.txt
Using grid catalog type: lfc
Using grid catalog : lfc.phy.bg.ac.yu
VO name: seegrid
Source URL: lfn:/grid/aegis/neda/wiki_acl/wiki_test
File size: 106
Source URL for copy: gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/aegis/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4.54540.0
Destination URL: file:/home/ivana/seegrid.txt
# streams: 1
# set timeout to 0 (seconds)
0 bytes 0.00 KB/sec avg 0.00 KB/sec inst
Transfer took 1010 ms
$ ll seegrid.txt -rw-rw-r-- 1 ivana ivana 106 Oct 23 17:21 seegrid.txt
The user successfully deletes and unregisters all replicas of the file.
$ lcg-lr -v lfn:/grid/aegis/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4 srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/filec7bee9b4-ba20-47a5-a86f-ce7045bcfa9e
$ lcg-del -a -v lfn:/grid/aegis/neda/wiki_acl/wiki_test VO name: (null) Using GUID : 6a14d029-6a6c-422a-934b-89eea3b76550 set timeout to 0 seconds srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4 is deleted srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/aegis/generated/2007-10-23/file7e370b0b-5e1c-465e-a738-a3fb7fb2ecb4 is unregistered srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/filec7bee9b4-ba20-47a5-a86f-ce7045bcfa9e is deleted srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/filec7bee9b4-ba20-47a5-a86f-ce7045bcfa9e is unregistered
Example 3: Manipulation with file on the Grid - different users, different VOs
Third example observes what happens when the user (ivana, AEGIS VO) that does not belong to the same VO, as the one (neda, SEE-GRID VO) that put file to the Grid, wants to manipulate with the file.
$ lfc-mkdir /grid/seegrid/neda/wiki_acl $ lfc-getacl /grid/seegrid/neda/wiki_acl # file: /grid/seegrid/neda/wiki_acl # owner: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka # group: seegrid user::rwx group::rwx #effective:rwx other::r-x default:user::rwx default:group::rwx default:other::r-x
$ lcg-cr -v --vo seegrid -d dpm.phy.bg.ac.yu -l lfn:/grid/seegrid/neda/wiki_acl/wiki_test file:/home/neda/dpmtest.txt
Using grid catalog type: lfc
Using grid catalog : lfc.phy.bg.ac.yu
Using LFN : /grid/seegrid/neda/wiki_acl/wiki_test
Using SURL : srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/seegrid/generated/2007-10-23/filedbdeecbd-f0eb-4e3b-ac33-199c2b5cea3e
Source URL: file:/home/neda/dpmtest.txt
File size: 106
VO name: seegrid
Destination specified: dpm.phy.bg.ac.yu
Destination URL for copy: gsiftp://dpm.phy.bg.ac.yu/dpm.phy.bg.ac.yu:/storage/seegrid/2007-10-23/filedbdeecbd-f0eb-4e3b-ac33-199c2b5cea3e.54605.0
# streams: 1
# set timeout to 0 seconds
Alias registered in Catalog: lfn:/grid/seegrid/neda/wiki_acl/wiki_test
106 bytes 0.21 KB/sec avg 0.21 KB/sec inst
Transfer took 1030 ms
Destination URL registered in Catalog: srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/seegrid/generated/2007-10-23/filedbdeecbd-f0eb-4e3b-ac33-199c2b5cea3e
guid:02f0db33-604b-448a-9b4d-3649b49481f4
Like in previous two examples user ivana can get information about file (lfc-ls -l, lcg-lr, lfc-getacl), create replica (lcg-rep), but there are problems when she wants to delete replicas. There are two replicas, original one, created by neda (SEE-GRID VO), on dpm.phy.bg.ac.yu and one created by ivana (AEGIS VO) on grid15.rcub.bg.ac.yu. Let's have look ACL of the file:
$ lfc-getacl /grid/seegrid/neda/wiki_acl/wiki_test # file: /grid/seegrid/neda/wiki_acl/wiki_test # owner: /C=RS/O=AEGIS/OU=Institute of Physics Belgrade/CN=Neda Svraka # group: seegrid user::rw- group::rw- #effective:rw- other::r--
User ivana tries to delete all replicas:
$ lcg-del -v -a lfn:/grid/seegrid/neda/wiki_acl/wiki_test VO name: (null) Using GUID : 02f0db33-604b-448a-9b4d-3649b49481f4 set timeout to 0 seconds httpg://dpm.phy.bg.ac.yu:8443/srm/managerv1: Permission denied srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/seegrid/generated/2007-10-23/filedbdeecbd-f0eb-4e3b-ac33-199c2b5cea3e is NOT deleted srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639 is deleted lfc.phy.bg.ac.yu: Permission denied srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639 is NOT unregistered lcg_del: Permission denied
As result of this operation one replica is deleted, the one created by ivana, and neither of them is unregistered, which can be verified by listing all replicas. The consistency is lost, catalog shows existence of non-existing replica.
$ lcg-lr lfn:/grid/seegrid/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/seegrid/generated/2007-10-23/filedbdeecbd-f0eb-4e3b-ac33-199c2b5cea3e srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639
Let's try to delete just replica on grid15.rcub.bg.ac.yu, first as ivana:
$ lcg-del -v -s grid15.rcub.bg.ac.yu lfn:/grid/seegrid/neda/wiki_acl/wiki_test VO name: (null) Using GUID : 02f0db33-604b-448a-9b4d-3649b49481f4 set timeout to 0 seconds httpg://grid15.rcub.bg.ac.yu:8443/srm/managerv1: No such file or directory srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639 is NOT deleted lcg_del: No such file or directory
and then as neda:
$ lcg-del -v -s grid15.rcub.bg.ac.yu lfn:/grid/seegrid/neda/wiki_acl/wiki_test VO name: (null) Using GUID : 02f0db33-604b-448a-9b4d-3649b49481f4 set timeout to 0 seconds httpg://grid15.rcub.bg.ac.yu:8443/srm/managerv1: No such file or directory srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639 is NOT deleted lcg_del: No such file or directory
Both cases show that there is no such file.
So, how to unregister an unexisting file? Command lcg-uf unregisters in the catalog a file residing on a SE. This command has to be used carefully. It just removes entries from the catalog, it does not remove any physical replica from the SE. Watch out for consistency!!! For this command GUID and SURL of the replica should be provided.
$ lcg-lg lfn:/grid/seegrid/neda/wiki_acl/wiki_test guid:02f0db33-604b-448a-9b4d-3649b49481f4
First try, as user ivana:
$ lcg-uf --vo aegis guid:02f0db33-604b-448a-9b4d-3649b49481f4 srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639 lfc.phy.bg.ac.yu: Permission denied lcg_uf: Permission denied
This user has no permissions for such action.
Second try, as user who put file to Grid originally (neda):
$ lcg-uf --vo aegis guid:02f0db33-604b-448a-9b4d-3649b49481f4 srm://grid15.rcub.bg.ac.yu/dpm/rcub.bg.ac.yu/home/aegis/generated/2007-10-23/file00b23d00-c814-4fe1-9190-22e7307b9639
went with a more success. Now just existing file is visible.
$ lcg-lr lfn:/grid/seegrid/neda/wiki_acl/wiki_test srm://dpm.phy.bg.ac.yu/dpm/phy.bg.ac.yu/home/seegrid/generated/2007-10-23/filedbdeecbd-f0eb-4e3b-ac33-199c2b5cea3e
