The implemented coarse grain parallelism exploits a parameterizable 2D decomposition in the x-y direction of the 3D physical domain (the vertical dimension is not decomposed). The automatic partitioning of the 3D domains generates by ComLib produces as many non-overlapping vertical rectangular beams as processors requested by the user. Each beam is then assigned to one processor of the parallel target computer. Some overlapping data-structure referred to as ``halo'' in the sequel are required; this enables to reuse all the computational routines existing in the sequential version of the Méso-NH code. Because most of the operators are discretized using 5 point stencil in the horizontal plane the ``halo'' have a width of only one cell but it may be easily increased.
The definition of the characteristics of the 2D decomposition and the communication informations is entirely managed the ComLib package and its related representation is stored in private data structure of the package. In this respect the subroutine INI_PARA_ll should be called before using any other functions of ComLib. To release these data structures at the end of the simulation, the function END_PARA_ll should be called.
In Figure 11 we depict an example of decomposition suitable for a parallel execution of Meso-NH on a two processor computer. Beam and are respectively allocated to processor 1 and 2; the variables associated with internal nodes in can be updated simultaneously provided that data in the halo (shadowed area in Figure 11) have been updated correctly prior to the computation. It is up-to the user to define the list of fields that should be kept updated in the halo region, through the ADD1DFIELD_ll,ADD2DFIELD_ll, ADD3DFIELD_ll API (Application Protocol Interface) subroutines of the ComLib library, as well as when this list of field should be updated using the UPDATE_HALO_ll, UPDATE_1DHALO_ll subroutines. A complete set of subroutines is available in ComLib to manipulate the list of fields (i.e. adding and deleting fields from a list).
The REMAP_2WAY_X_ll, REMAP_X_2WAY_ll, REMAP_X_Y_ll, REMAP_Y_X_ll subroutines described in this user guide should be useless for regular Meso-NH users. These communication functions have been implemented to parallelize the FLAT_INV Meso-NH subroutine that implements a fast Poisson solver used as preconditioner for the Pressure iterative solver. Their description in this document only aim at helping future maintenance of the FLAT_INV subroutine that is used as a ``black-box'' by regular Meso-NH users.
Here is an example showing how the routines can be used for parallelizing Meso-NH :
PROGRAM EXAMPLE USE MODE_ll USE MODD_ARGSLIST_ll, ONLY TYPE(LIST) USE MODD_PARAMETERS USE MODD_DIM$n USE MODD_LBC$n USE MODD_NESTING INTEGER :: IINFO_ll ! return status of the // routines REAL, DIMENSION(:,:,:) :: XRUS,XRVS ! 3D fields TYPE(LIST) :: TPFIELDSLIST ! list of 3D fields ! ! set the specifications of the simulation (dimensions, LBC type, ...) ... ! CALL SET_JP_ll(JPMODELMAX,JPHEXT,JPVEXT) CALL SET_DIM_ll(KIMAX_ll, KJMAX_ll, KKMAX) ! call made by the first model ! CALL SET_LBX_ll(HLBCX(1), KMI) ! call made by each model KMI CALL SET_LBY_ll(HLBCY(1), KMI) CALL SET_XRATIO_ll(NDXRATIO_ALL(KMI), KMI) CALL SET_YRATIO_ll(NDYRATIO_ALL(KMI), KMI) CALL SET_XOR_ll(NXOR_ALL(KMI), KMI) CALL SET_XEND_ll(NXEND_ALL(KMI), KMI) CALL SET_YOR_ll(NYOR_ALL(KMI), KMI) CALL SET_YEND_ll(NYEND_ALL(KMI), KMI) CALL SET_DAD_ll(NDAD(KMI), KMI) ! CALL INI_PARA_ll(IINFO_ll) ! initialise data structures used in the parallel routines ! CALL INIT ! initialise MesoNH variables ! .... ! ! dimensions of the extended 2way subdomain for the local processor. CALL GET_DIM_EXT_ll('B',IIU,IJU) IKU=NKMAX +2*JHVEXT ! ALLOCATE XRUS(IIU,IJU,IKU) ! Allocate memory on the extended domain ALLOCATE XRVS(IIU,IJU,IKU) ! Allocate memory on the extended domain ! ! Construction of the list of fields to be communicated ! CALL ADD3DFIELD_ll(TPLOCALLIST_ll, XRUS) ! add field XRUS to list CALL ADD3DFIELD_ll(TPLOCALLIST_ll, XRVS) ! add field XRVS to list ! ! ! calculation of source terms ...... ! !!!! PRESSURE SOLVER ! ! Update the halo for local variables ! (of the fields included in the list TPLOCALLIST_ll) ! CALL UPDATE_HALO_ll( TPLOCALLIST_ll, IINFO) ! ! Example of modification of the list ! CALL DEL3DFIELD_ll(TPLOCALLIST_ll, XRUS) ! delete field XRUS from list CALL DEL3DFIELD_ll(TPLOCALLIST_ll, XRVS) ! delete field XRVS from list ! CALL END_PARA_ll(IINFO_ll) ! END PROGRAM EXAMPLE
Here is an example showing how the parallel remapping routines are used in FLAT_INV in the parallel Poisson solver :
SUBROUTINE FLAT_INV(...) ! !!**** *FLAT_INV * - Invert the flat quasi-laplacian operator ! ! PY RHS of the equation ! PF_1_Y Solution of the equation ... ! USE MODE_ll ! !* 0.1 declarations of arguments ! ... ! !* 1. COMPUTE LOOP BOUNDS ! ------------------- ! ... ! ! Get the local dimension ! CALL GET_PHYSICAL_ll(IIB,IJB,IIE,IJE) CALL GET_DIM_EXT_ll('X',IIX,IJX) CALL GET_DIM_EXT_ll('Y',IIY,IJY) ! ! Allocate the arrays for x-slices and y-slices decompositions ! ALLOCATE(BAND_X(IIX,IJX,IKU)) ALLOCATE(BAND_Y(IIY,IJY,IKU)) ALLOCATE(BAND_YR(IIY,IJY,IKU)) ! !* 2. COMPUTE THE ARRAY INCREMENTS FOR THE FFT ! ---------------------------------------- ! ... ! !* 3. FORM HOMOGENEOUS BOUNDARY CONDITIONS FOR A NONCYCLIC CASE ! ... ! ! first step : every sub-domains updates its boundaries ! ... ! ! second step : 2way structure -> xslice structure, + data shift ! CALL REMAP_2WAY_X_ll(PY,BAND_X,INFO_ll) ! !* 4. APPLY A REAL TO COMPLEX FFT ! --------------------------- ! ... ! ! Perform FFTx for BAND_X ! ... ! ! Transposition X-> Y ! CALL REMAP_X_Y_ll(BAND_X,BAND_Y,INFO_ll) ! ! Perform FFTy for BAND_Y ! ... ! !* 5. MATRIX INVERSION FOR THE FLAT OPERATOR ! -------------------------------------- ! Result in BAND_YR ... ! !* 6. APPLY A COMPLEX TO REAL FFT ! --------------------------- ! ! -1 ! Perform FFTy for BAND_YR ! ... ! ! Transposition Y-> X ! CALL REMAP_Y_X_ll(BAND_YR,BAND_X,INFO_ll) ! ! -1 ! Perform FFTx for BAND_X ! !* 7. RETURN TO A NON HOMOGENEOUS NEUMAN CONDITION FOR NON-CYCLIC CASES ! ----------------------------------------------------------------- ! !* ! ! 7.1 Transposition -> 2way ! CALL REMAP_X_2WAY_ll(BAND_X,PF_1_Y,INFO_ll) ! !* 7.2 complete the lateral boundaries of PF_1_Y ! ... ! DEALLOCATE(BAND_X) DEALLOCATE(BAND_Y) DEALLOCATE(BAND_YR) ! END SUBROUTINE FLAT_INV !