Using MPI with PETSc library
From EGEE-see WIki
Contents |
Introduction
MPI, as a way of communication between processes, can be used for implementing programs that work with matrices or systems of equations. The programmer is responsible for synchronization between processes by explicitly calling MPI functions. It is time consuming and distracts the programmer from thinking about the actual problem that is being solved. It such a scenario, lot of time and effort goes on the implementation of communication and parallelization process.
When solving scientific problems that concern matrices, vectors and mathematical systems of equations in general, there is other way available. Instead of programming on MPI level, it is possible to increase level of abstraction by using Portable, Extensible Toolkit for Scientific Computation (PETSc). It is a library that hides communication details by eliminating MPI calls from program. Communication and parallelization is done internally by PETSc and is invisible for the programmer.
We compared the execution time of two solutions for the same problem, implemented using MPI directly and using MPI with PETSc. The comparison shows that PETSc solution has an order of magnitude faster execution, although the actual improvement depends on size of problem being solved and experience of MPI programmer. Based on this experience, it seems that PETSc’s usage of MPI is very efficient and therefore is hard to match by a programmer with other primary focus. Since PETSc deals with problem partitioning and processing and data distribution, it is very suitable for “accidental” programmers.
PETSc Overview
PETSc (Portable, Extensible Toolkit for Scientific Computation) is a package incorporating parallel data structures and routines to numerically solve partial differential equations (PDE) and related linear and non-linear problems.
In other words, PETSc is suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations (PDE).
Copyright of PETSc belongs to University of Chicago. It is open sorce project and can be downloaded free of charge. Permission to copy and modify this software and its documentation is granted if the original copyright notice is retained.
Architecture
PETSc employs the MPI standard for all message-passing communication. But it hides details of communications within objects. User orchestrates communication at the higher level of abstraction then direct MPI calls. Basically, it eliminates MPI from MPI programming!
PETSc promotes object oriented programming. Design is not based on data, but instead on operations you perform with or on the data. For example, a vector is not a one dimensional array of numbers but an abstract object where addition and scalar multiplication (and many more) are available.
In this manner, all
- vector-vector operations,
- vector-matrix operations and
- matrix-matrix operations
are already defined on vector and matrix objects.
It integrates BLAS (Basic Linear Algebra Subprograms) library, which can independently used for solving linear algebra problems.
It, also, integrates LAPACK (Linear Algebra PACKage) for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems.
The linked picture shows the internal structure of PETSc: (PETSc Architecture)
Features
PETSc is intended for use in large-scale application projects, and several ongoing computational science projects are built around the PETSc libraries. In reality there is no limitation in size of problem and the number of processors used. It has done the problems with 500 million of unknowns on 6,000 processors.
PETSc is easy to use for beginners.
PETSc includes an expanding suite of parallel linear and nonlinear equation solvers that are easily used in application codes written in C, C++, Python, and Fortran. PETSc provides many of the mechanisms needed within parallel application code, such as simple parallel matrix and vector assembly routines that allow the overlap of communication and computation. In addition, PETSc continuously improves support for distributed arrays. The list of available features:
- Parallel vectors
- scatters
- gathers
- Parallel matrices
- several sparse storage formats
- easy, efficient assembly.
- Scalable parallel preconditioners
- Krylov subspace methods
- Parallel Newton-based nonlinear solvers
- Parallel timestepping (ODE) solvers
- Complete documentation
- Automatic profiling of floating point and memory usage
- Consistent interface
- Intensive error checking
- Portable to all systems (Linux, Windows, Apple, IBM, Sun, HP)
- Limited parallel grid/data management
- Over one hundred examples
- Usable from Fortran, C, C++ and Pyton
- PETSc is supported and will be actively enhanced for the next several years.
For detailed information on PETSc, please consult the PETSc documentation page.
Example of usage
#include petscksp.h
int main(int argc,char **argv){
KSP solver;
PC prec;
Mat A;
Vec X,B;
MPI_Comm comm;
PetscInt i,its,n=10;
PetscErrorCode ierr;
// Initialisation of PETSc (and MPI environment in background)
PetscInitialize(&argc,&argv,0,0);
comm = MPI_COMM_WORLD;
// Creation of mathematical system (matrix A, rhs vector B and solution vector X)
// PETSc is doing memory allocation internaly
MatCreate(comm,&A);
MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);
VecCreate(comm,&B);
VecSetSizes(B,PETSC_DECIDE,n);
VecDuplicate(B,&X);
// Initialization of matrix and vector elements
for (i=0; i<n-1; i++) {
MatSetValue (A,&i,&i,i*10,INSERT_VALUES)
VecSetValue (B,&i,1,INSERT_VALUES);
}
// Assembling of matrix and rhs vector
MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
VecAssemblyBegin(B);
VecAssemblyEnd(B);
// Creation of CG linear solver with JACOBY preconditions
KSPCreate(comm,&solver);
KSPSetType(solver,KSPCG);
PCSetType(prec,PCJACOBI);
// Solver call
KSPSolve(solver,B,X);
// Getting number of iteration done
KSPGetIterationNumber(solver,&its);
printf("\nConvergence in %d iterations.\n",(int)its);
// Destruction of allocated objects
MatDestroy(A);
VecDestroy(X); VecDestroy(B);
KSPDestroy(solver);
// Finalization of PETSc (and MPI environment in background)
PetscFinalize();
return(0);
}
Resources
PETSc download, installation guide, user manuals, tutorials, and much more are available on PETSc’s official site http://www-unix.mcs.anl.gov/.
