Managing Sets of Files and Replicas Within LFC Catalog

From EGEE-see WIki

Jump to: navigation, search

This Wiki page is a part of SEE-GRID Gridification Guide. It is contributed by Belgrade University Computer Centre.

Contents

[edit]

Overview

Tool for managing file sets and replicas in LFC Grid Catalog supports additional operations over the files and logical structures on gLite grid middleware, with purpose of maintaining file replica sets, file sets, and managing usage of storage elements. Its implementation is based on SEE-GRID File Management Java API.

This tool can be used from both UI and grid jobs. Both command-line interface and Java API are provided. Although the primary goal of this tool is to help users and developers in managing their data, it can be also used or recommended by site administrators when a SE is decommissioned.

It is compatible with LCG 2.7.x and gLite grid middleware.

Available features

  • Replication or migration of files and directory trees within VO between Storage Elements (e.g. for SE decommissioning). If source pathname is directory, operation can be performed recursively.
  • Copying of files and trees between LFC and local filesystem. Data migration between VOs or users by using local filesystem as temporary storage. There are four types of migration:
  • From LFC to local filesystem
  • From local filesystem to LFC
  • From LFC to LFC
  • From local file system to local filesystem

If source pathname is directory, operation can be performed recursively.

  • Moving of files and directories - like copying, but with removing of source files and emptied directories after successful copying of each item
  • Purging of directory trees by deleting all files from specified pathname or only the files modified before given date
  • Removing excess replicas from SE given in a specified list or not belonging the list, based on a limit on the number of replicas


Planned features

  • Integration with existing applications
  • User-friendly GUI
[edit]

Installation and Configuration

The following APIs should be already installed and configured by your system administrator. This is done by following the gLite installation guides 1, 2. These APIs come with gLite middleware.

  • GFAL C/C++ API
  • lcg_util C/C++ API
  • LFC C/C++ API

Download the Tool for managing file sets and replicas in LFC Grid Catalog. It also contains SEE-GRID File Management Java API and Replica Management Utility Scripts for command line execution. Unpack the archive into a directory present within CLASSPATH environment variable, or add the target directory to CLASSPATH. Optionally, make scripts from rmu-scripts subdirectory available in PATH.

  • For all operations, the virtual organization name value is obtained from -vo input parameter. If -vo parameter is not set, the used virtual organization is obtained from the "repmngr.properties" file in the current folder. If properties file or VO value in the file is missing, the value from environment variable LCG_GFAL_VO is used.
  • The host name and port number of BDII being used to obtain the list of available storage elements can be set in properties file. If not, the value of LCG_GFAL_INFOSYS environment variable is used as default.

Please note that the usage of GFAL related operations, i.e. file content transfers between a storage element and local filesystem require LCG_GFAL_INFOSYS to be consistent with the value in properties, if any.

Properties file configuration:

  • Set the value of property VO to your Virtual Organization name (for example: VO = seegrid)
  • Set the value of properties bdii.host.name and bdii.host.port to point to the BDII
  • Optionally, set the value of property SE.ignore.list - the list of SEs to be ignored when obtaining available SEs from the BDII. Use space to separate list items. (example: SE.ignore.list = se1.ignoredhost.org se2.ignoredhost.org)
[edit]

Documentation

Javadoc for the Tool for managing file sets and replicas in LFC Grid Catalog

[edit]

Downloads

version 1.0 (released 11/06/2007)

ZIP archive with source code, classes, properties file, and scripts (114 KB)

[edit]

User Guide

[edit]

Replication or migration of files and directory trees within VO

This operation supports the user to do replication or migration of files and directory trees from one SE to another. The operation is used in the cases when is necessary for the replica to be on the certain SE or for SE decommissioning. Parameters:

  • sourcePathname – name of file or directory in LFC Grid Catalog on which the operation is performed.
  • isRecursive - if set and SourcePathname is directory, its subdirectories should be iterated as well.
  • maxReplicaCount – if final number of replicas would be above this limit, no replication is done, ignored if <1 or if migration is performed.
  • freedStrorageElement – if set, the replica at this SE will be deletes after generating new one at target SE, effectively migrating the replica
  • targetStorageElement – replication/migration target

ReplicaManagementUtil class method signature:

public static void replicationOrMigration(String srcPathname, boolean isRecursive, int maxRepCount, 
                                          String freedSE, String targetSE, String vo)

Script usage from command line: 

rmu-replicate-or-move <soucePathname> <targetSE> [-m MaxRepCount] [-r] [-f freedSE] [-vo VO]
[edit]

Copying of files and trees between LFC and local filesystem

This operation supports the user to copy files and trees between LFC and local filesystem. It can be also used to migrate data between VOs or users by using the local filesystem as temporary storage. Parameters:

  • sourcePathname – name of file or directory in LFC Grid Catalog or local filesystem on which the operation is performed.
  • destinationPathname - name of target file or directory tree in LFC Grid Catalog or in local filesystem
  • isRecursive - if set and SourcePathname is directory, its subdirectories should be iterated as well.

ReplicaManagementUtil class method signature:

public static void copyingBetweenLFCAndLocalFilesystem(String srcPath, String destPath, boolean isRecursive, String vo);

Script usage from command line: 

rmu-copy <sourcePathname> <destPathname> [-r] [-vo VO]
[edit]

Moving of files and trees between LFC and local filesystem

This operation supports the user to move files and trees between LFC and local filesystem. It’s used to when migrating data between VOs or users by using local file system as temporary storage. Parametars:

  • sourcePathname – name of file or directory in LFC Grid Catalog or local filesystem on which the operation is performed.
  • destinationPathname - name of target file or directory trees in LFC Grid Catalogue or in local filesystem
  • isRecursive - if set and SourcePathname is directory, its subdirectories should be iterated as well.

ReplicaManagementUtil class method signature:

public static void movingBetweenLFCAndLocalFilesystem(String srcPath, String destPath, boolean isRecursive, String vo);

Script usage from command line: 

rmu-move <soucePathname> <destPathname> [-r] [-vo VO]
[edit]

Purging of directory trees in LFC

This operation removes the whole directory tree or purges old files in it. If date and time are set, the operation deletes only files files with earlier LFC timestamps. Parameters:

  • pathname – name of file or directory in LFC Grid Catalog or local filesystem on which the operation is performed.
  • date or yyyy, mm, dd, hh, mm, and ss - date and time.

ReplicaManagementUtil class method signatures:

public static void purge(String pathName, String vo);
public static void purge(String pathName, Date date, String vo);
public static void purge(String pathName, int year, int month, int dayOfMonth, String vo);
public static void purge(String pathName, int year, int month, int dayOfMonth, int hourOfDay, int minute, int second, String vo);

Script usage from command line: 

rmu-purge <pathname> [-DATE yyyy mm dd [hh mm ss]] [-vo VO]
[edit]

Removing excess replicas

This operation is maintaining and relieving the system with deleting certaing number of replicas in order, firstly from SE located on RemoveFromStorageElements lists, than randomly among other elements, except replica from SE located on KeepOnStorageElements lists, until the number of replicas is reduced to given value. Parametars:

  • Pathname – name of file or directory trees in LFC Grid Catalogue or in local filesystem on which the operation is performed.
  • IsRecursive - – if Pathname is directory, check if subdirectories should be iterated as well.
  • MaxReplicaCount – number of limit of the replicas.
  • RemoveFromStorageElements – list of SEs from which replicas should be removed first
  • KeepOnStorageElements – list of SEs from which replicas should not be removed

Method signature: 

public static void removeExcessReplicas(String pathname,boolean isRecoursive, int maxRepCount,
                                             String[] remSE, String[] keepSE, String vo)

Usage for command line (do not use spaces within -rem and -keep paremeters): 

rmu-remove-excess-replicas <pathname> [-m MaxRepCount] [-r] [-rem se1[,se2...]] [-keep se3[,se4...]] [-vo VO]
[edit]

Examples

The classes in package yu.ac.bg.rcub.grid.dataManagement.util.cli are good examples of the usage of the API provided by yu.ac.bg.rcub.grid.dataManagement.util.ReplicaManagementUtil class. They are available in Downloads section.

[edit]

Contact

Nebojsa Djuric (nebojsa.djuric AT rcub.bg.ac.yu)

Retrieved from "http://wiki.egee-see.org/index.php/Managing_Sets_of_Files_and_Replicas_Within_LFC_Catalog"
Personal tools