Summary

The coupler drives the whole coupled model, ensuring the synchronisation of the different component models and the exchange of the coupling fields directly between the components or via additional coupling processes. When needed, the coupler performs transformations on the coupling fields. Another important part of the coupler is the model interface library linked to each component model which interfaces it to the rest of the coupled model. As I/O and coupling data share many characteristics, it was decided to develop one common model library for both purposes.

The different constituents of the PRISM coupler and I/O library are therefore the Driver, the Transformer, and the PRISM System Model Interface Library (PSMILe). The PSMILe includes the Data Exchange Library, which performs the exchanges of coupling data, the I/O library, and some coherence check and local transformation routines.

In this paragraph, the PRISM coupled model high level architecture is first presented. The functionalities of each constituent and their priority of development are detailed in the second section.

Outline

1. Coupled model high level architecture
2. Detailed functionalities for the PRISM coupler and I/O library
  2.1. General requirements
  2.2. Driver functionalities
  2.3. Transformer functionalities and parallelisation
  2.4. PSMILe functionalities
        2.4.0 PSMILe general characteristics
        2.4.1 Data Exchange Library (DEL) functionalities
        2.4.2 I/O library
        2.4.3 Coherence check routines
        2.4.4 Local transformation routines

1. Coupled model high level architecture
 

An overview of a coupled model is presented here. The following graphical view details the different parts of the system :

The elements of a coupled model are the following:

• The Driver, which monitors the whole coupled model.
• The Transformer separate entity T, which performs transformations on the data.
• The component models Mi, Mj, or Mk, interfaced to the rest of the coupled model through the PRISM System Model Interface Library (blue and purple squares).
• The Potential Model Input and Output Description (PMIOD): a container describing the relations the model is able to establish with the rest of the coupled model through inputs and outputs.
• The Specific Model Input and Output configuration (SMIOC): a container describing the relations the model  will establish with the rest of the coupled model through inputs and outputs for a specific experiment.
• The Specific Coupling Configuration (SCC): a container describing all activated coupling fields for a specific experiment.
• The files, containing data.

The different elements a coupled model are detailed hereafter by describing the three basic phases of its construction and execution:

• Definition of the entities to be coupled (component models, coupler elements, files,...)
• Composition of these entities in a coupled system
• Deployment of the coupled system onto a set of computing resources

In the definition phase, the different elements of the coupled system are prepared:

• The component models (Mi, Mj, or Mk), including the PRISM System Model Interface Library (PSMILe) :

Each component model has to include specific PRISM System Model Interface Library (PSMILe) instructions that will allow the component model to interact with the rest of the PRISM System at run-time. The PSMILe, represented here by the blue and purple squares, includes the Data Exchange Library, which performs the exchanges of coupling data directly between the component models or between the component models and other coupling processes, the I/O library, and some coherence check and local transformation routines.
 

• The Potential Model Input and Output Description (PMIOD):

For each model, the Potential Model Input and Output Description (PMIOD) describes the relations the model is able to establish with his external environment. The PMIOD contains a short description of the model, its grids, and the list of all data requested or produced by that particular component model and their description (the metadata). The input and output data can be divided into 3 categories.
 

• Transient input and output variables: data evolving during the run, received or provided at run-time by the model at an a priori unknown frequency, from or to an external entity (another model -coupling variables- or a disk file -I/O variables). For example, coupling data and diagnostics belong to this category.

 
• Restart variables: data requested initially by the model to (re)start the simulation and provided previously by the model itself (not by an external entity). These data are saved to disk at regular intervals during each run. It remains to be clarified if restart variables should be treated differently from transient input and output variables. From the discussions, it now emerges that both WP4a and WP3a tend to agree that there is no reason to treat restart variables differently from transient input and output variables. The restart characteristic could just be indicated as an additional metadata in the PMIOD.

 
• Persistent input parameters: data requested initially by the model to define the physical configuration of the experiment. These parameters may have default values but may be adapted by the user. For coupled models, the "universal parameters" are parameters that require a consistent definition among all the model components.


Each model and its respective PMIOD is made available to potential PRISM users by the model administrator.
Note: one code may implement different interfaces under certain conditions. But one PMIOD is not related to one implementation (one executable)
 

• The input files:

All input files containing data required for the run have to be generated.

• The coupler Driver and Transformer separate entity:

The Driver, which monitors the whole coupled simulation, and the Transformer separate entity, which performs required transformation on the data, have to be available.

In the composition phase, a particular user assembles a particular coupled model.

Selection of component models:

The user first chooses the component models he wants to couple for one particular experiment.
 

Input file selections

The user selects the input files containing information that will be used during the simulation, such as forcing fields.
 

Driver and Transformer separate entity selection:

The user selects the PRISM Driver and Transformer separate entity.
 

Constitution of each model Specific Model Input and Output Configuration (SMIOC):

Based on each model PMIOD, the user generates for each model a Specific Model Input and Output Configuration (SMIOC). The SMIOC describes the relations the model will effectively have with his external environment through inputs and outputs for a specific experiment.

For transient input and output variables, the user may decide that a particular data will 1- have no role in the simulation, 2- be read from a file or written to a file (I/O data), or 3-be exchanged between to component models (coupling data). For I/O data, the user indicates in the SMIOC, the name(s) of the respective file(s), the input or output frequency, and possibly the local and non-local transformations required on the data (see 2.3). For coupling data, the user just refers to the Specific Coupling Configuration (SCC).
Note: The Transformer will never perform directly the reading/writing of a transient input/output variable in/to a disk file. Pseudo-model performing the I/O will have to be written for transient input or ouput I/O variables requiring transformation(s) to be performed by the Transformer. These I/O  variables will therefore become coupling variables exchanged between pseudo-models and "real" models, and will be treated as such.  Like a "real" model, each pseudo-model will come with its PMIOD. The only case for which the Transfomer may be asked to performed directly an I/O is to read a  field that  it will  combine with a standard coupling field. The separation coupling-variable-info-in-SCC vs I/O-variable-info-in-SMIOC is therefore OK.

For restart variables, the user is only allowed to indicate the name of the restart file and, possibly, the restart saving frequency. However, this last parameter should have the same value for all component models and should therefore be treated as a universal parameter in the SCC.

The value of persistent input parameters are read at run-time in the SMIOC. The user may be allowed to change the default value therein. For persistent input parameters which are also universal parameters, the value taken into consideration is the one indicated in the SCC.
 

Constitution of the Specific Coupling Configuration (SCC):

The user constitutes only one Specific Coupling Configuration (SCC) for each particular coupled model simulation. The SCC centralises the description of all activated coupling fields and all related coupling parameters chosen by the user (source and target models, coupling frequencies, local and non-local transformations, etc.) for one particular experiment. The SCC also contains the universal parameters prescribed by the user.
SL 13/06/02: However, parameters like gravitational constant are also called universal constants. Perhaps I am wrong? If not: For these parameters I prefer a f90-module solution where they are declared either by '
REAL PARAMETER :: g=9.81 or by '
REAL           :: g=9.81
depending on the case, and where this module MUST (coding rule) be USEd in the model (or IS USEd in PRISM_Init). This of course means that recompiles are necessary when we
calculate the climate on jupiter. However, to pass e.g. g by the coupler is overdoing to my feeling.

These universal parameters will be present in each model SMIOC and in the SCC. In stand-alone mode, the appropriate PSMILe function will automatically read these information in the SMIOC. In non stand-alone mode, the PSMILe will automatically request the information from the Driver who will have read the information in the SCC. One universal parameter is the initial date of the simulation. Another universal parameter is the initial date of the run, as one simulation may have to be split into many runs to fit the limits of  he job queuing system. The PRISM System will automatically chain the different runs and will automatically change the value of the initial date of the run in the SCC or in the SMIOC. Note that this is different from what was concluded at the wp4b meeting in May.

It is proposed that the PMIOD, SMIOC, and the SCC containers be implemented as  XLM files.

Note: Some deployment information will have to be in the SCC.

See also http://www.cerfacs.fr/PRISM/MTCI/notes_PSMILe_V1.html

SV 28/10/2002:
Usually one code represents one component model (ocean, biogeo, atmos, ice, land, atmos chemistry) and will come with one PMIOD.
However two component models (e.g. ocean + biogeo) could be assembled into one code which would them come with two PMIODs. In that case, there will be two SMIOCs and each component model would have to call prism_init_comp with its own "model_name" as argument.
It could even be that two components, formally defined as such in PRISM,  be assembled into one code which would come with only one PMIOD in which all Potential Input and Output of both components would be described. In that case, there will be one related SMIOC and all processes of that code would have to call the prism_init_comp with the same "model_name" as argument. In other words, a prism_init_comp with one particular "model_name" refers to one particular SMIOC.
As soon as coupling relation(s) have to be established between different PMIODs, whether these PMIODs are associated to the same code or not, a SCC will be created and the PRISM Driver will be required.
If forcing data need interpolation by the Transformer before being used by a component model, then a pseudo-model reading these data will have to be present; in this case, the component model and the pseudo model will come with their PMIOD which will be used to write the SMIOCs and one SCC (the Driver is required). If the forcing data need no interpolation, the PSMILe will read directly the data in the files. In this case, there is only one PMIOD, and
therefore one SMIOC (no SCC nor the Driver are required.)
We could therefore have the following situations (and many others, of course!):
-One code, two component models, two PMIODs -> two SMIOCs, no coupling relation between the component, therefore no SCC, no Driver.
-One code, two component models, two PMIODs -> two SMIOCs, coupling relations between the components, therefore a SCC and a Driver.
 

At run-time, the different parts of the system will play different roles. A more detailed description of the functionalities of each constituent is presented in part 2.

The Driver: launches the component models, monitor their execution and termination.

The Transformer separate entity T: performs required transformations on the I/O and coupling data.

 

The PRISM System Model Interface Library (PSMILe):

The PSMILe includes the Data Exchange Library, which performs the exchanges of coupling data directly between the component models or between the component models and the separate Transformer entity, the I/O library, and some coherence check and local transformation routines. At run-time, specific PSMILe instructions will perform the following actions:

Initialisation:

Declaration of PSMILe internal data structure.

Message passing initialisation.

I/O initialisation.

Initialisation of persistent input parameters, read directly in the SMIOC.

Initialisation of universal parameters, either received from the Driver or read directly in the SCC.

 

Metadata declaration and initialisation:

 

Definition of the metadata describing input or output data (for example the grid coordinates, mesh areas, mask, partitioning), and definition of associated identificators.

 

Declaration of transient and restart variables:
 

Association to the relevant metadata identificators (see below).

Access to user-defined data information: for each data declaration, the PSMILe consults the SMIOC and identifies the user's choice for that particular experiment (coupling or I/O data , input or output frequency, source and target models, source or target file, transformations, etc.)

 

Sending and receiving of data
 

The actions performed by the PSMILe below each sending or receiving instruction depend on the user's choices read in the declaration phase in the SMIOC and in the SCC: the library may simply return, or perform local transformations, and/or perform the exchanges between the models , and/or perform the reading or writing into files, etc.

Coupling termination

 
All actions related to finalizing the run.

2. Detailed functionalities for the PRISM coupler and I/O library

As detailed above, the different constituents of the PRISM coupler are: the Driver; the Transformer; and the PRISM System Model Interface Library (PSMILe), linked to the component models and which interfaces the component model with the rest of the coupled model. The PSMILe includes the Data Exchange Library, the I/O library, and some coherence check and local transformation routines.

For each of these constituents, the list of possible requirements established in the REDOC II.2 paragraph was revised and choices of functionalities that should be implemented in the different versions of the PRISM coupler were made, considering the answers to the REDOC I.4 template. These choices are detailed below.

For each functionality, a priority of implementation is given: "1" means that the functionality should be provided for the PRISM coupler first version (D3a1, month 12),  "2" for PRISM coupler second version to be used in the demonstration runs (D3a2, month 24), and "3" means that the functionality may be provided for the PRISM coupler final version (D3a3, month 36).

Note: See also REDOC I.3.2

2.1. General requirements

The overhead associated to the global system modularity and flexibility is acceptable. (2, 3)

The whole system is portable and efficient on the different hardware architectures used for climate modelling, on dedicated or shared hardware resources. Standard and portable solutions should be preferred. However, for critical issues  for which a portable solution would not exist or would lead to very low efficiency, machine dependent options could be offered. (3)

The design and implementation lead to code easy to maintain and can be easily modified to support future model or coupling functionalities. (2, 3)

Design reflects a clear separation of responsibilities for the different parts of the coupler. (2, 3)

The PRISM System infrastructure can be used to technically assemble a coupled system based on any component models, even if these models do not conform to the PRISM physical interfaces given that they include the PRISM System Model Interface Library. (1, 2, 3)

The PRISM System infrastructure can be used to couple an arbitrary number of component models; any component can be one-way or two-way coupled with any other component. (1, 2, 3)

2.2. Driver functionalities
 

 The Driver manages the whole coupled application. It launches the component models, monitor their execution and termination, centralise and distribute universal parameters which require a consistent definition among all component models, and centralize and distribute information on the component model status during the simulation.

The driver could keep a central role during the whole simulation and manage also the exchanges of coupling data. The preferred design option here is to decentralize the coupling functionalities as much as possible in the Data Exchange Library and in the Transformer, and therefore to reduce as much as possible the role of the Driver. This option is probably applicable only for static coupled simulations and allows an easier evolution toward heterogeneous coupling (different component models running on different machines).

As detailed below, the choice of a static Driver was also made. The workload of a static driver is likely to be small, even more if the decentralizing option is followed. The Driver could be one separate process used only for it, but could also sit in one separate coupling process used also for the separate Transformer entity, or even could be part of the PSMILe master process of a master model started by the user initially. The first two options are still open regarding the Driver implementation.

Model execution and control:

• The Driver manages static simulations (with respect to the process management) : all component models are launched initially and run for the entire length of simulation. Full MPMD launching by the Driver (with MPI2) and MPMD initial starting (with MPI1) are supported. (1, 2, 3)
The Driver will not manage dynamic model execution (one or more models starting and ending during the simulation), neither conditional model execution (one model is started during the simulation only if a particular scientific condition is met).
WP3h considered the dynamic model execution functionality as essential to run time-slice experiments. Other groups considering this functionality as desirable mentioned that it could be useful to run alternatively different coupling configurations (e.g. for asynchronous coupling). We consider here that these requirements can be fulfilled by chaining different static simulations.
Technically, the argumentation presented in REDOC II.2 Annexe I was reviewed and it was evaluated that the disadvantages linked to a dynamic configuration were more important than its advantages.
Therefore, the choice was made to design a static, less sophisticated, but more efficient Driver.
• The Driver can start a global coupled system flexible in terms of executables (extreme are: each component is a separate executable -MPMD-, or all components run in parallel or in sequence within only one executable -SPMD). If the SPMD mode is chosen for two or more components, the developer will be responsible assembling for the components into one executable. (2, 3)
• The Driver can give some statistic on the load balancing of the run. (3)
• The Driver includes a timing functionality which can sample with identical absolute time the duration of events for all component models. (3)
• Based on efficiency criteria, the Driver decides where each local transformation will be performed (in the PSMILe or in the Transformer).

 

• The global model input parameters are parameters that needs to be consistently defined in the coupled system (initial date, length of integration, calendar, earth radius, restart saving frequency, etc.). This information is defined by the user in the a coupled model configuration file. The driver gets these parameters in the configuration file and, on a PSMILe request, transfers this information to each component model.  (For stand-alone model, the model PSMILe should read directly the information in the coupled model configuration file.) (1, 2, 3)
• The driver does not centralize all model information (grid definition, distribution, etc.). Each model PSMILe is be responsible for transferring the appropriate information to the appropriate processes. (2, 3)
• Depending on a log level chosen by the user in the coupled model configuration file, internal PSMILe log functions or log functions implemented by the developer in the code are activated. The log function transfers information on the state of the model to the driver which centralizes this information and may then transfers this information to a higher level controlling layer or to the user. (3)

• In a decentralized and static approach, the matching between output coupling data produced from one model and input coupling data requested by another model could be performed initially and the exchange should be managed at run-time by the Data Exchange Library included in each model PSMILe. The other option is that the driver performs the matching and, at run-time, manages the exchanges of coupling data based on this matching. The decentralized option is the preferred one but the two options are still open now. The matching is based on user's choices indicated in the coupled model configuration file; the way the matching and the exchanges are managed are, in any case, transparent for the developer and for the user.

• If one component aborts, the whole simulation must shut down cleanly. (1, 2, 3)
The coupled model will most probably be a MPI1 or MPI2 application. In that case, if one component model aborts, the whole MPI job terminates automatically and the Driver can have no control on the termination. However, in the case of PSMILe exceptions (the error code returned by one PSMILe routine indicates an error), the model developer may decide to make the model to abort; the routine called in that case first interacts with the Driver which then performs appropriate actions (such as saving the most recent cached information), sends a message to the model which then aborts, and terminates the coupled application. (3)
• The driver constantly updates, by writing in a restart log file or by any other equivalent mean, the last date for which all model restarts were saved. This information is sent to the driver by each model PSMILe. (2, 3)
• Neither the driver nor any external controlling instance automatically restarts a coupled system after an unforeseen termination (machine breakdown, ...). This is not desirable because the diversity of faults is large and in many cases, human intervention is mandatory before the simulation should be restarted (ex: the disk on which the diagnostics are saved is full).
• The driver will not be able to shutdown the simulation cleanly if a specific scientific condition is met (e.g. average SST exceeds some predefined value). This functionality will be fulfilled by a conditional PRISM_Abort implemented in the code by the model developer or by a conditional PRISM_Abort managed automatically by the PSMILe (see below).

see also plan_inter_230402.html

This paragraph first gives some definitions. In the second section, the preferred design options for the PRISM coupler Transformer location and parallelisation are presented. In the third section, an exhaustive list of transformations and grids on which these transformations should be performed is presented, together with other specific requirements, and associated priority and calendar.

2.3.1 - Definitions

• Point-wise transformation: an operation that can be completed on each grid point without any external information, neither from the model neighbouring grid points, neither from another model, such as time averaging or addition of coupling fields given on the same grid
• Local transformation: an operation that can be completed in a model without any information from another model, such as finding the maximum value of a field.
• Non-local transformation: an operation that requires information  from another model, such as interpolation.

Location for the different types of transformations

The preferred design option is the one in which non-local transformations are performed in the separate Transformer entity (T), as they require information coming from different models. Point-wise and local transformations will be workable in the PRISM Model Interface Library (PSMILe) linked to the model before sending or after receiving the data. However, point-wise and local transformations will also be available in the separate Transformer entity T, for example, to combine coupling fields coming from different source models after their interpolation on the target grid.

The same rules apply for two component models assembled into one executable: all point-wise and local transformations will be performed directly in the PSMILe, while the data will have to be treated by the separate Transformer entity T if non-local transformations are required. This last case however is not likely to happen, as two components assembled into one executable will in most cases share the same grid and same partitioning.

Ideally, the choice of whether the transformation is performed by the PSMILe or in the separate Transformer entity T should be decided automatically by the coupler and this should be transparent for the user (3).

Note 04/2002: For coupling variables, the list of transformations will be listed in the SCC by the user. Non-local transformation will necessarily be performed by the Transformer. Local transformations can be performed in the PSMILe or by the Transformer; for each simulation, the Driver will decide where they will be performed, based on efficiency criteria.

Transformer parallelisation

As detailed above, the transformation routines included in the PSMILe will perform local transformations, and not only point-wise transformations; their full parallelisation is therefore required when the PSMILe is linked to a fully parallel component model (3).

Non-local transformations will be performed in the separate Transformer entity T. Different options of parallelisation are possible. The "one-executable full parallelisation" option presented in REDCO II.2, Section 3.3 is the preferred one (3). A fall back solution would be a simpler parallelisation of the separate Transformer entity T as one executable with openMP.

Note on I/O 04/2002: The Transformer will never read/write directly a transient input/output variable in/to a disk file. Pseudo-model performing the I/O will have to be written for transient input or ouput I/O variables requiring transformation(s) to be performed by the Transformer. These I/O variables will therefore become coupling variables exchanged between pseudo-models and "real" models, and will be treated as such. Like a "real" model, each pseudo-model will come with its PMIOD. The only case for which the Transfomer will perform directly an I/O is to read a field that  will  be combined with a standard coupling field.

List of transformations

A list of relevant transformations is given hereafter. For each transformation, it is specified whether the transformation is "point-wise", "local" or "non-local".

• 1D, 2D, and 3D spatial interpolations

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

All these transformations are non-local.
 

• S1 - Nearest-neighbour interpolation function:

For each target grid point, the n nearest neighbours on the source grid, weighted or not by their distance, are averaged.
• S2 - Nearest-neighbour Gaussian weighted interpolation function:

For each target grid point, the n nearest neighbours on the source grid, weighted by the value of a Gaussian function at their distance from the target point are averaged.
• S3 - 1st order interpolation function:

Standard 1st order bilinear interpolation.
• S4 -  2nd order interpolation function:

Standard 2nd order bicubic interpolation.
• S5 - First order conservative remapping:

This scheme will guarantee that the line (1D)-/area (2D)-/ volume (3D)-integrated field (e.g. water or heat flux) is conserved between the source and the target grid.
• S6 - Second order conservative remapping:

This scheme will also guarantee that the line (1D)-/area (2D)-/ volume (3D)-integrated field (e.g. water or heat flux) is conserved between the source and the target grid.
• S7 - Remapping using user-defined remapping info (ex: MOZAIC)

• Other 1D, 2D, and 3D spatial transformations

• S8- Conservation:

This non-local operation ensures global energy conservation between source and target grids (ex: CONSERV).
• S9- Combination or merge:

This local and point-wise operation combines different parts of different coupling fields or of other predefined external data given on the same grid (ex: FILLING). This operation may involve the smoothing of the fields near the different domain borders; in that case, the operation is still local but not point-wise.
• S10- Masking:

With this local and point-wise operation, only the points listed in index have meaningful data and the others are changed to missing (ex: MASK). It should be possible to use another coupling field (e.g. sea-ice concentration) for masking (cf mail R. Doescher, GLOBC/COUPLE/oasis_2.4 , 10 Oct 2000).
• S11- Scattering:

This local operation scatters the model data onto the points listed in an index.
• S12- Gathering:

This local operation gathers from the input data all the points listed in an index.
• S13- Collapse:

This local operation results in the collapse of any dimension or combination of dimensions by various, possibly weighted, statistical operations, such as mean, max, min, etc. (possibly relative to a threshold, e.g. maximum of positive values). (ex: CHECKIN, CHECKOUT)
• S14- Subspace:

This local operation results in the extraction of subspaces or hyperslabs in any combination of spatial dimension.
• S15- Algebraic operations:

Local and point-wise operations, such as addition, subtraction, multiplication, etc., with possibly different coupling fields or predefined external data (given on the same grid) and numbers as operands (+, -, X, SQRT, ^2, SIN, LOG, ...) (Ex: BLASOLD, BLASNEW, SUBGRID, CORRECT)
• S16a - 1st order  extrapolation function
• S16b - 2nd order extrapolation function:

This transformation is required to address the specific problem of the wind curl along the coast.
• Time operations
• T1- Time integration, average, variance, extrema, linear interpolation

Local and point-wise operations.

The following grids should be supported for the above scheme. These grids have the following common characteristics:

• The grids may have masked grid points.
• The grids may have "holes" (i.e. they do not cover the whole sphere, e.g. regional grids).
• The grids may be global or regional (except H3 reduced grid which is always global).
• The grids may have overlapping grid points (except H3).

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

2D grids
 

• H1 - Lat-lon grids:

The grid is given by the intersection of meridians and parallels. Each (i,j) grid point can be described by the value for the indices i and j of two 1-D arrays, latitude(j) and longitude(i).
• The latitudinal mesh sizes can be regular or irregular, i.e. latitude(j) is or not constant.
• The longitudinal mesh size can be regular or irregular,  i.e. longitude(i) is or not constant.
• The grid may overlap in longitude with N overlapping grid points.
• The grid may have grid points at the pole and/or at the equator.
• The border of the meshes can be approximately placed half-way between two grid points, but the exact location of the border meshes should be given with the grid.
• H2 - Cartesian and stretched and/or rotated grids (logically rectangular):

Each (i,j) grid point can be described by the value for the indices i and j of two 2-D arrays, latitude(i,j) and longitude(i,j).
• The grid may overlap itself in the i and/or j direction.
• The exact location of the border meshes should be given with the grid.
• H3 - Reduced grids

The grid is composed of a certain number of latitude circles, each one being divided into a varying number of longitudinal segments. The grid can be described by the number of latitude circles, N_lat, the latitudinal position of each circle npos(N_lat) and by an 1-D array giving the number of longitudinal segments for each latitude, n_seg(N_lat). The total number of grid points, N_tot, is the sum of all elements of n_seg. The grid can also be described by two 1-D arrays, latitude(N_tot) and longitude(N_tot). This grid may be considered as one particular case of unstructured grids
• There is no overlap of the grid.
• There is no grid point at the equator nor at the poles.
• There are grid points on the Greenwich meridian.
• The exact location of the border meshes should be given with the grid.
• H4 - Unstructured grids

The grid, with N_tot number of grid points, has no logical structure.  The grid must be described by two 1-D arrays, latitude(N_tot) and longitude(N_tot)
• The grid may overlap itself for some grid points.
• There may be a grid point at the equator or at the poles.
• There may be grid points on the Greenwich meridian.
• The mesh can have an arbitrary number of sides. The exact location of the border meshes should be given with the grid.
3D grids
 
• V1 - Reproduction of the same horizontal grid at different levels

The same horizontal grid is reproduced at different vertical levels. Each level has its particular mask.
The vertical levels can be:
• V1-1 : given at regular or irregular depth or height levels (z co-ordinate)
• V1-2: hybrid: first level follows the topography (atmosphere models or the bathymetry (ocean models), last level follows an isobar(atmosphere) or the surface (ocean), progressive transition in between.
• V1-3 : given at regular or irregular isopycnal (density) levels (r co-ordinate)
• V2 - Different horizontal grids at different levels

The horizontal grid is not reproduced at different vertical levels. The horizontal grid can be rotated, translated, or totally unstructured.

• To support scalar coupling data. (For priority see below).
• To support vector coupling data in the standard spherical geographical coordinate system. (For priority see below).
• To support vector coupling data in any set of local coordinate system. (For priority see below).
• To support fields with undefined variables. (2, 3)
• To support coupling fields which characteristics may change over time as simulation develops , e.g. grid, resolution, distribution (CF dynamic simulation).(3)
• To be able to save, at a user-defined frequency, its restart data (e.g. time accumulated data). (2,3)
• To support source and target coupling domains that totally or partially overlap (e.g. global atmosphere with a regional ocean, regional model nested into global model, etc.). (1, 2, 3)
• In debug mode (specified by the user in the SCC), to dump the coupling variables just after receiving and just before sending.
• To have an entry point for user-defined transformations.

The following paragraph gives the priority of development (1, 2, or 3 -the meaning of each number is given in the introductory paragraph of section 2) for the different transformations on the different grids listed above. When two numbers are given, it means that parts of the functionality will be provided for the respectives coupler versions.

Transformations on 2D scalar coupling fields

Transformations on 2D vector coupling fields
 
• Transformations on 2D vector coupling fields given in the standard spherical coordinate system

Transformations S1, S2, S3, S4, S6, S7, S9, S10, S11, S12, S13, S14, S15, S16a, S16b, T1 are given the same priority than for 2D scalar fields.
Transformations S5, S8 are of priority 3.
• Transformations on 2D vector coupling fields given in any set of local coordinate system

Transformations S1, S2, S3, S4 for H1 and H2, S7, S10, S11, S12, S15, S16a, S16b for H1 and H2 are of priority 2.
Transformations S4 for H3, S5, S6, S8, S9, S13, S14, S16b for H3 are of priority 3.
Transformations on 3D coupling fields
 
• Interpolations for V1-1 and V1-2 grids

For V1-1 and V1-2 grids, the treatment of 3D coupling fields will be addressed for the second PRISM coupler version. The idea is to proceed with a multiple 2D interpolation (i.e. 2D interpolation on many horizontal or hybrid levels) plus a simple linear interpolation vertically (priority 2). The priority given to the multiple 2D interpolations or extrapolations (S1, S2, S3, S4, S5, S6, S7, S16a and S16b) for scalar or vector fields for the different horizontal grids is 2, or more if a lowest priority is given above for the equivalent (single) 2D interpolation scheme.
• Other transformations for V1-1 and V1-2\

For V1-1 and V1-2 grids, the local and point-wise transformations S10, S15, and T1 are given priority 2.
For V1-1 and V1-2 grids, transformations S8, S9, S11, S12, S13, S14 are given priority 3.
• Interpolations and other transformations for V1-3 and V2 grids

These transformations will not be addressed within PRISM 3-year project.
Transformations on 1D coupling fields
 
All transformations are given priority 3

2.4. PSMILe functionalities

The  PRISM System Model Interface Library (PSMILe) is the set of routines implemented in a component model code to interface it with the rest of the coupled model. The classes of PSMILe instructions that will be  invoked in the component model code at run-time are described in Section 1, C - Deployement phase. Here, the functionalities of the different PSMILe constituents (i.e. the Data Exchange Library, the I/O library, and the coherence check and local transformation routines) are presented in mode details.

2.4.0 PSMILe general characteristics

• The modifications to implement in the model code are as reduced as possible. (1, 2, 3)
• The PSMILe is layered, and complexity is hidden from the component code. (1, 2, 3)
• Interface routines once defined and implemented are not (or as little as possible) subject to modifications  between the different versions of the PRISM coupler. However new routines may be added. (1, 2, 3)
• A good trade-off is chosen between (I) a concise list of parameters for each subroutine call (more subroutines provided with a shorter list of parameters) and (II) a small number of subroutines, each one having a longer and more complex parameter list. The complexity arises from the need to transfer not only the coupling data but also the meta-data. (1, 2, 3)
• The component models are able to run in a stand-alone model without modifications, with or without an external driver. (2, 3)
• The description of the data, i.e. the meta-data (e.g. units, grid coordinates, mask, distribution, ...), and the model information (e.g. length of a timestep) is given by the model through the PSMILe and not duplicated externally by the user (2, 3). This information may change during the simulation. (3)
• The PSMILe is extendable to new types of coupling data (e.g. data given on arbitrary grids). (1, 2, 3)
• The PSMILe includes the Data Exchange Library and the I/O library as the most external layers. The PSMILe therefore automatically manages the cases for which input coupling data are not provided by another model but have to be read into a file; this is transparent for the component model. The format of these data files could be a standard PRISM fixed format (2). At a later stage, different formats could be supported for these data files, implying that the I/O instance can interpret their content.(3)
• The PSMILe includes some local transformation routines and performs the transformation required locally before the exchange with the rest of the PRISM System. (2, 3)
• The PSMILe performs some checks of coherence on coupling and I/O data, according to a coherence check level defined by the user in the coupled model configuration file. (3)
• In debug mode (specified by the user in the SCC), the PSMILe should be able to dump the coupling variables just before leaving or just after entering the model.
• RV 18/06/02: We should think about packed storage schemes.
• See also http://www.cerfacs.fr/PRISM/MTCI/notes_PSMILe_V1.html
• RV 02/07/02: include files which are used for PSMIle internal purposes should have the prefix psmile_ and include files which can be include by model code shoud have the prefix prism_.
• For PSMILe interface, see http://www.cerfacs.fr/PRISM/MTCI/comments_PSMILe/comments_PSMILe.html and http://www.cerfacs.fr/PRISM/MTCI/grid_def.html

2.4.1 Data Exchange Library (DEL) functionalities

The Data Exchange library (DEL) performs the exchanges of coupling data between the component models, or between the component models and the separate transformation entity. The DEL must therefore be included as the most external layer in the PSMILe.

Data transfer between separate processes will be implemented using the message passing interface MPI, which is a widely used and portable standard. MPI implementations completely supporting the MPI standard are available for every architecture used by the climate modelling community either as open source public domain code or as proprietary software optimised and installed on high performance computer system. Furthermore MPI is best suited for the close coupling between separate processes, as in climate system modelling, since individual MPI implementations are designed to use the most efficient network on a specific architecture.

Since all parallel climate model codes support communication via MPI the introduction of alternative approaches like CORBA requires additional software like Fortran ORBs.  Another possibility is wrapping the Fortran codes using a C++ ORB which can require major changes to the involved Fortran codes as well.  (For experiences gained with wrapping Fortran code see http://accl.grc.nasa.gov/IPG/CORBA/wrap_fortran.html).

In addition, alternative approaches such as CORBA handle data transfer via TCP/IP, which is not well suited for a fast and efficient parallel data transfer. MPI processes in contrast may communicate simultaneously without interfering the communication of other processes, while the same kind of communication will cause conflicts on a TCP/IP connection. Transfer rates between two processes can differ by a factor 10^5 to 10^6 when comparing CORBA with MPI. Furthermore a complete CORBA standard is not available for every architecture.

The DEL detailed functionalities are:

• The exchange can occur directly between two component models without going through additional transformation processes. When the component models are parallel and have different data partitioning, repartitioning associated to direct communication is required; all type of distributions usually used in model component codes are be supported. (2 for static simulations, 3 for CF dynamic simulations)
• "End-point" data exchange: when producing coupling data, the source model does not know what other model will consume it; when asking for coupling data a target model does not know what other model produces it. (1, 2, 3)
• Coupling data produced by a source model can be consumed by more than one target model. (2, 3)
• Occurrence of the exchange can be different for the different coupling fields. (1, 2, 3)
• For each coupling field, the exchange occurs at a fixed frequency for the whole simulation . (1, 2, 3)
• For each coupling field, the exchange may occur at different fixed frequencies for different periods. (2, 3)
• Coupling data produced from one model at a particular time may be required as input coupling data for another model at another time. The DEL can take into account a timelag defined by the user in the coupled model configuration file. (2, 3)
• Conditional occurrence of an exchange, depending on parameters dynamically calculated during the simulation, will not be supported within PRISM 3 years.
• The coupling data can be of different types: integer and real, complex (32, 64, 128 bits) appearing as multidimensional arrays, but without time dimension (2,3). For structures, operators and functions, other PSMILe primitives will be defined when required, but not within PRISM 3 years.
• The coupling data characteristics, and therefore the associated metadata, may change over time as the simulation develops e.g. the grid or the resolution (CF dynamic simulations). (3)
• Coupling data produced by one model may be only partially consumed by the target model; extraction of subspaces, hyperslabs or indexed grid points may be required before the exchange. (2,3)
• Sending and receiving instructions can be placed anywhere in the source and target code and possibly at different location for the different coupling fields. (1, 2, 3)
• The DEL insures that component models can be executed concurrently, in a regular sequence (one after the other), or in some pre-defined combination of these two modes. (1, 2, 3)
• The DEL offers efficient data exchange implementations for loose and strong coupling. Loose coupling is the configuration in which the two component models are run sequentially or concurrently as two separate executables. Strong coupling is the configuration in which the two component models are run within the same executable. (2, 3)
• The DEL could perform the matching between output coupling data produced by one model and input coupling data requested by another model (see related discussion above in the Driver section) (1, 2, 3)
• The DEL should perform automatic type conversion if the source model and target model coupling field types do not match ???

For the glossary: a "partition" is the set of grid points treated by one process. One partition may be composed of one or several segments, or of one or several blocks.

Each model process will initially define (or re-define during the run) its grid and partition by calling one PRISM_DefineBlock2D (or PSMILe_def_horigrid) function. The partition will be described in a vector of integers, indexList. PRISM_DefineBlock2D (or PSMILe_def_horigrid) will return a handle for the grid and partition, gridId (or hori_grid_id).

Now the question is: which type(s) of partition should be supported and therefeore possibly described in indexList.

In OASIS, 3 types of partitioning are supported (see documentation p.49-50-51, http://www.cerfacs.fr/globc/software/oasis/doc_oasis2.4.ps):
  0) Serial: no partitioning
  1) Apple: each process partition is composed of one segment
  2) Box: each process partition is composed of one block
  3) Orange: each process partition is composed of many segments.

In Palm, 2 types of partitioning are supported (see http://www.cerfacs.fr/globc/PALM_WEB/doc/external/QIKGUIDE/qikgu260.html):
  0) SINGLE_PROC: no partitioning
  1) scaLAPAK REGULAR: each process partition is composed of many blocks of same shape
  2) CUSTOM: each process partition is composed of many blocks, each one having possibly its own shape.

I think that Rene and I agreed that the PSMILe should at least support the CUSTOM partition, including a description of each block halo (this is not included in Palm). This means that one element of indexList has to be the number of blocks; the description of each block, given by the tupple "i-halo-low,i-halo-high,j-halo-low,j-halo-high,i-comp-low,i-comp-high, j-comp-low,j-comp-high" will then be the other elements of indexList. (If more than one type of partition is supported, the first element of indexList should be the type of partition.)

One detail: I think that there is no need to associate one handle per block, as practically all blocks of one partition will be transfered at once. If this is not the case, more than one grid/partition could be defined by one model process (each grid/partition would then be composed of the blocks beeing transfered at once)..

Should other types of partition be supported?

Now, concerning repartitioning (transfering the data between two models which partitionings do not match):

I would like to draw your attention, that all calculations needed for repartitioning between Apple <-> Serial, Box <-> Serial, Orange <-> Serial are already in Oasis (easy, I know!). More important: all calculations needed for repartitioning between two CUSTOM partitionings that do not match are already in Palm (not so easy!). These calculations are described in the document that you will find here attached. Please note that this document was written some time ago and that things slightly evolved since then but it can give you a first overview.

2.4.2 I/O library

The I/O library performs the exchanges with files stored on disk. Activated variables, regional selection, temporal and geographical transformation, and file names are chosen by the user in the SMIOC (see Section 1, B - Composition phase). Metadata and run time information are provided at run-time by the component model through the PSMILe. The data and the associated metadata will be read or written to the disk files.

For data access, calls to the NetCDF (http://www.unidata.ucar.edu/packages/netcdf/) library will be implemented.  Support of formats other than NetCDF will not be implemented, but entry points for reading and writing other file formats will be provided.

Execution on parallel machines will have to work efficiently. MPI-IO is the standard solution and will be evaluated. In a first step, we will avoid parallel I/O by doing regional selection for input data and by doing postprocessing operation after a simulation to combine multiple outputs files provided by a parallel execution.

SL, 29/04/2002:  Several runs make up an experiment and I would require that the partitioning of the experiment into runs is transparent to the user. This is in addition to the fact that the content of all output files be independent of the partitioning (restartability of all components).  I consider it desirable e.g., that the 'output period' of data contained in one model raw output file (when it is archived) (and not the length of a run) is part of the experiment definition. This is easy to accomplish with GRIB data (just concatenate when an output period' is finished).  With NetCDF is is possible, but more complicated.  This can be important for the ease of use of the data diagnostics.

Restart files:  -If the user wants to restart its coupled model after an unforeseen termination, he/she will have to look up in the "restart log file" the last date for which all model restart were saved and indicate it as the starting date in the coupled model configuration file (SCC). At the beginning  of each run (yes, I think "run" is appropriate here!), the driver automatically reads the initial date in the SCC and transfers this information to the model. When the model calls the PSMILe routine that reads in the restart, this date could be used to check if it reads the correct restart (the date of the restart could be in the file name, but more preferabily in the file header, so that a check of coherence is possible when reading the file).

2.4.3 Coherence check routines

The PSMILe will perform some checks of coherence on coupling and I/O data, according to a coherence check level defined by the user in the Specific Coupling Configuration (SCC) file. The coherence check instance will:

• Understand some standard conventions of metadata. (3)
• Based on the metadata, perform compatibility checks between data produced by a source component and its description established by the model developer in the PMIOD. (3)
• Based on metadata description, perform compatibility checks between data produced by a source component and data required by a target component. (3)
• Based on metadata description, recognize type of coupling data (flux, vector, scalar) and verify that the user's transformation choice is appropriate. (3)
• Warn the user if the coupling and I/O frequencies are not synchronized. (3)
• Automatically check an abort condition defined by the user in the SCC or in the SMIOC on the value of the input and output data, and perform the abort if the condition is met. (3)

2.4.4 Local transformation routines

The PSMILe will perform the following transformations locally. The priority for including these local transformations in the PSMILe is given here :

• Combination of different data produced by one model (S9) (2).
• Masking (S10) (2).
• Scattering  (S11) (3).
• Gathering (S12) (3).
• Collapsing (S13) (3).
• Subspace (S14) (3).
• Algebraic operations (S15) (2).
• 1st and 2nd order extrapolation (S16a, S16b) (3).
• Time integration, average, variance, extrema, linear interpolation (T1) (2).

2.4.5 Error handler

• informs the driver about PSMILe internal failures like failed assertions
• does not overwrite the default Un*x signal handling or the MPI default error handling
• provides also routines to interpret error codes into error messages

3. Model requirements
 

• a model switch to tell it to start from restart files, (indication in the output listing that you did !!)
• ability to use calendar and all universal parameters defined by the user in the SCC