next up previous contents
Next: 3. Interface Routines: ComLib Up: mainexpand Previous: 1. Introduction   Contents

Subsections

2. Flow structure

2.1 Temporal loop

The parallelization of the code has been achieved in modifying the temporal loop to take into account the data domain decomposition. Fig. 3 shows which routines of code have been modified.

Figure 3: Flow structure for the temporal loop
\begin{figure} \centerline {\psfig{file=Figures/modeln2.eps,width=13.cm}}\par\end{figure}

Three types of routines can be distinguished: routines to communicate fields between processors, routines allowing the treatment of physical boundary conditions and routines doing global operations. Others communications are required for distributed I/O and pressure solver and are described in other sections.

2.1.1 Communication of fields between processors

These exchanges are necessary to update the halo of all subdomains (routine UPDATE_HALO_ll) and to also update the second layer of the halo (routine UPDATE_HALO2). Communications concerning the second layer of the halo are connected to the 4th order numerical diffusion.

The following routines have been modified to include resfreshemnt of the halo:

2.1.2 Informations on the physical boundaries

Modifications in some Meso-NH routines have been brought to allow to identify the location of physical boundaries. The ComLib routines LNORTH_ll, LWEST_ll, LSOUTH_ll and LEAST_ll are called in Meso-NH routines ANTI_DIFF, BOUNDARIES, DFLUX_CORR, MPDATA, NUM_DIFF, PHYS_PARAM$n, RAD_BOUND, and TURB. In addition the routine GET_INTERSECTION_ll is called in RELAXATION to allow to know the relative location of the relaxation region.

2.1.3 Global operations

Specific global functions (MIN_ll and SUM3D_ll) have been thus implemented in
RESOLVED_CLOUD routine. Other global operations are made in the Meso-NH model in the statistical analysis of Meso-NH (BUDGET and LES_SPECTRA parts) and will be further modified.

2.2 LB fields

2.2.1 What are the LB fields?

Figure 4: Organisation of the South-North LB arrays (for relaxation along the y-axis) in the non-parallelized code (a); Pannel b is an example of decomposition in sub-domains.
\begin{figure} \begin{center} \epsfig {file=Figures/FigLB1.eps, width=17cm} \end{center}\end{figure}

There are two sets of LB fields : The LB fields on the western and eastern sides (for relaxation in the x-axis direction) and the LB fields along the southern and northern sides (for relaxation in the y-axis direction). We will discuss the problem of distribution of the LB fields on the southern and northern sides. It can be transposed to the LB-fields on the western and eastern sides.

The LB fields for relaxation along the y-axis are constitued of two slices along the South and North borders (Fig.4a). In the non-parallellized version of the model, the two slices are stored in a single array of dimension (IIU_ll,2*NRIMY +2,:); the southern slice is stored in the section (1:IIU_ll,1:NRIMY+1,:) and the northern slice, in the section
(1:IIU_ll,NRIMY+2:2*NRIMY +2,:). NRIMY is the depth of the riming zone. In the FM-files, the two slices of the LB fields can be larger than that is needed in the model. The LB arrays are of dimension (1:IIU;2*ILBSIZEY+2,:) in the FM-file (see Fig.4a).

For the LB fields of variables located at a mass grid point or a u-grid point, the two slices correspond to the areas (1:IIU_ll,1:1+NRIMY) and (1:IIU_ll, IJU_ll - NRIMY: IJU_ll ) on the global domain reference (Fig.4b) . For the LB fields of variables located at a v-grid point, the two slices correspond to the areas (1:IIU_ll,2:2+NRIMY) and
(1:IIU_ll, IJU_ll - NRIMY:IJU_ll ) on the global domain. If there is no relaxation, the LB fields are stored in a single array of dimension (IIU_ll,2,:) for variables located at a mass grid point or a u-grid point, and of dimension (IIU_ll,4,:) for variables located at a v-grid-point. Table 1 gives for all the possibilities considered in the model the location of the LB arrays in the global domain reference.

Fig 4b shows an example of decomposition in sub-domains : some sub-domains have empty intersection with the LB areas while others contain one or two pieces of the LB areas. Note that for domains with no-empty intersection with both the southern LB area and the northern area, the sizes along the x axis of the two intersections are the same, as we have rectangular sub-domains.

2.2.2 The distribution of the LB fields

The distribution of the LB fields operates as following :

  1. Allocation the LB arrays (in INI_MODELn) :

    Figure 5: Determination of the sizes of the South-North LB arrays on each sub-domain for variables at mass points and with relaxation
    \begin{figure} \begin{center} \epsfig {file=Figures/FigLB2.eps, width=17cm} \end{center}\end{figure}

    The routine GET_SIZEY_LB (GET_SIZEX_LB for west-east borders) provides the sizes of the LB arrays; given the location of the LB areas in the global domain reference (see Table 1), the location (IXORI:IXENDI,IYORI:IYENDI) of the intersections with the sub-domain in the local sub-domain reference is obtained by calling the interface routine GET_INTERSECTION_ll (Fig. 5). The size along the y-axis IJSIZEYF of the LB arrays is the size of the intersection of the sub-domain with the southern LB area, plus the size of the intersection of the sub-domain with the northern LB area. The sizes of the LB array can be zero if the two intersections are empty. Table 1 gives the sizes of the LB arrays for all the possibilities considered in the model.

    Figure 6: Initialisation of the LB arrays
    \begin{figure} \begin{center} \epsfig {file=Figures/FigLB35.ps} \end{center}\end{figure}

  2. Initialisation of the LB arrays :

    New global variables NSIZELBX_ll, NSIZELBY_ll, NSIZELBXU_ll, NSIZELBYV_ll, NSIZELBXvar_ll and NSIZELBYvar_ll (var being R, TKE, EPS,SV) are initialized in INI_MODELn and stored in the module MODD_CPL; they are the total depths of the western and eastern LB areas and the total depths of the northern and southern LB areas for the total domain. Their values depend of the on/off relaxation switch (see Table 1). For the NSIZELBXvar_ll and NSIZELBYvar_ll, the on/off activation of the variable var is also taken into account.

    The LB fields (or sources) can be initialized at three locations in the model : at the level of READ_FIELD for the initial time, at the level of INI_CPL for the first coupling file, and at the level of LS_COUPLING for the other coupling files. The routines SCALAR_LS_COUPLING and UVW_LS_COUPLING have been replaced by the routine LS_COUPLING in the parallel version. The routines READ_FIELD, INI_CPL and LS_COUPLING have also been reorganized in the parallel version in order to share the common code which has been grouped in the routines INI_LB and INI_LS.

    In the routine INI_LB, the sizes of the LB areas in the FM-file and over the global domain are retrieved. Then, each LB fields are initialized by successive calls to the routine FMREAD_LB. Inside this routine, the LB fields are read in the FM-file and stored temporary in an array over the global domain (Step 1 in Fig. 6). Then, the distribution of the fields over each subdomain is performed : the local coordinates of the intersection of the LB areas with the subdomain are retrieved by successive calls to GET_INTERSECTION_ll (Steps 2ab in Fig. 6); then, the corresponding section in the temporary LB array over the global domain is determined; and finally, the I/O processor sends the corresponding section to each processor and initializes its own section (Step 3 in Fig. 6).


    \begin{sidewaystable} % latex2html id marker 123\begin{center} \begin{tabular... ...iable on the grid and to the relaxation switch.} \end{center}\end{sidewaystable}

2.3 INIT

2.3.1 Introduction

Figure 7: INIT
\begin{figure} \begin{center} \epsfig {file=Figures/init1.eps} \end{center}\end{figure}

Figure 8: INI_MODELn
\begin{figure} \begin{center} \epsfig {file=Figures/ini_modeln.eps} \end{center}\end{figure}

To parallelize the INIT block, different possibilities have been explored. The specific management of the maximum memory space during a parallel job have imposed the choice. This allocation is required once at the start of a job and no desallocation is possible during the life of the job. So INIT and the model temporal loop must use quite the same memory space.

The data domain decomposition is performed in INIT and all the routines work on sub-domain in a parallel way. The processings on global domain are exceptional (eigen value of the flat Laplacian operator in trid)

For the parallelization implementation, INIT can be divided in three parts (see Fig.7).

At the end of INI_MODELn, the update of the halos for the arrays listed in 2D-list and 3D-list are done. So all the arrays are correctly initialized over the extended sub-domain of each process at the end of INI_MODELn.

2.3.2 Detailed description of the modifications

Only the main modifications are described below. We don't detail the modifications linked to the I/O parallel package; a list of examples for use the I/O routines can be found in the guide elsewhere in this book.

2.4 BUDGETS

Two options are available for the budgets :

The results are computed in the local domain of each processor. Just before writting, the local results are reduced on the global domain with the END_CART_COMPRESS and END_MASK_COMPRESS routines.

Figure 9 shows the flow diagram after parallelisation. The modifications are now described in detailed for each routine :

Figure 9: Tree diagram of budget routines.
\begin{figure} \begin{center} \epsfig {file=Figures/budget8.ps,height=18cm} \end{center}\end{figure}

next up previous contents
Next: 3. Interface Routines: ComLib Up: mainexpand Previous: 1. Introduction   Contents
serveur WWW de Meso-NH
2001-11-15