Minutes of WP3a Technical Meeting

List of participants:

NEC-CCRL: Hubert Ritzdorf, Rene Redler
NEC-ESS: Thomas Schoenemeyer, Kolja Kuse
M&D (MPIM): Stephanie Legutke, Hannes Thiemann
SGI GmbH: Reiner Vogelsang
FECIT: Jean Latour
CERFACS: Samuel Buis, Damien Declat, Etienne Gondet, Thierry Morel,  Andrea Piacentini,  Sophie Valcke

Summary:

General definitions, news and ideas from PRISM  were presented.

A list of possible requirements for the PRISM coupler was reviewed. It was decided that each participant should rank these requirements in terms of priority from a user's point of view and/or in terms of difficulty from a developer point of view.

The two coupler developed at CERFACS were presented: PALM , a tool to couple data, models and data assimilation techniques in a modular, flexible and dynamic way , and  OASIS  , the coupler of geophysical codes widely used in the climate modelling community.

Developments foreseen for OASIS within PRISM first year and  First suggestions for the Model Technical Coupling Interface  both of which will constitute the WP3a first deliverable (D3a1), were discussed. It was suggested that a meeting should be arranged with the WP2c and WP4a groups to evaluate how far the model  interfaces for coupling data and other I/O data should be harmonized.

Finally,  the different  WP3a tasks  were distributed among the partners.

General definitions, news and ideas from PRISM

1. General definitions:

1.1 Coupler:

 A coupler is composed of three functional parts, assembled into libraries attached to the model (including the Model Technical Coupling Interface) and/or of additional coupling processes:

• the controller/driver which launches the models, monitors their execution,  and controls the whole simulation.
• the communicator which performs data exchanges with appropriate data repartitioning if needed.
• the transformer: performs all required transformations on the coupling  fields.

1.2 Model Coupling Interface:

The term "Model Coupling Interface" gathers 3 different concepts in PRISM

Model Physical Coupling Interface: nature of coupling information  exchanged between any two components (WP 3b-c-d-e-f-g-h)

Model Algorithmic Coupling Interface: flow chart of exchange (WP 3b-c-d-e-f-g-h)

Model Technical Coupling Interface: set of routines the user has to call in his model to perform the coupling (WP 3a)

1.3 Data repartitioning:

Action of transferring and rearranging coupling data between two models which data distributions in parallel processes do not match.

2.1 PRISM System Specification Committee

Following the PRISM Les Diablerets workshop in June, informal electronic discussion led to the formation of an unofficial PRISM system specification committee (PRISM SSC) which should be formally elected at the kick-off meeting. This committee met at IPSL in Paris on September 27th, 2001. The PRISM SSC suggested that before any line of code is written in the PRISM project, the requirements, constraints, architecture and design of the system need to be carefully defined; this is precisely the purpose of the workpackage 2a.

The PRISM SSC therefore proposes that the specifications of the PRISM System be organized into two documents. The first document called "Requirements, Design Options and Constraints" (REDOC) should define both the "ideal" PRISM System and the existing constraints. REDOC should be ready at the end of project month 3 and will help make choices for the actual work undertaken within the PRISM project. This work will be described in the second document "Architecture Choices, Detailed Design and Implementation" (ARCDI). A first version of ARCDI should be ready at the end of project month 6. The detailed content of this second document could still evolve after project month 6 but not its general principles and main lines. A PRISM System Specification Handbook - Scoping phase  detailing the content of REDOC is under preparation and should be discussed at the kick-off meeting.

REDOC will contain the list of the coupler requirements for an "ideal" PRISM System (see REDOC paragraph I.4). Based on this paragraph, we (WP3a) will then have to define possible options for the PRISM coupled system architecture (see REDOC paragraph II.1). The discussions reported here below can be considered as a first foundation stone in the REDOC construction. Later, we will have to describe in ARCDI which coupler functionalities will effectively be developed and how these will be implemented within the 3 years of PRISM.

2.2 A possible PRISM configuration

A possible and probably ideal configuration of a PRISM coupled model was presented (see figure here below). This configuration is based on different discussions that occurred so far in PRISM. It was presented for discussion and is not intended to fix any of the PRISM coupler functionalities; as said above, these will be determined with the whole PRISM group in the first 6 years of the project.
 
 

 

At T1, the beginning of the simulation, the driver D launches only the ocean O and the atmosphere A; O is distributed on 2 processes (two boxes for O), A on 3 processes (3 boxes for A). As there is no Land Scheme (LS) and therefore no correspondent coupling information produced, the coupler reads automatically the equivalent information in files (in black).

At time T2, when the simulation has reach some kind of equilibrium (defined for example by the drift of SST which is calculated by O and given by O to the driver D), the driver D launches also an atmospheric chemistry model (AC), a Land model (LS), a Sea Ice model (SI), and an Ocean Biogeochemistry model (OB). AC, LS, SI, OB are respectively distributed in 3, 3, 2 and 2 processes.
-A and LS have same grid and same distribution: the communication is direct (c to c).
-A and AC have same grid and same decomposition: no interpolation nor repartitioning is required. Furthermore, they run sequentially by construction. The user decides to make only one executable out of A and AC (one big blue box): between A and AC the exchange is not done by message passing but by memory sharing for efficiency.
-A and O have different grids and different decompositions: interpolation and repartitioning are required; as the number of fields exchanged is small, the user decides to perform the interpolation on the atmosphere processors (I attached to A), and the repartitioning is automatically performed by the coupler communication library (c).
-A and SI have different grids and different decompositions: interpolation and repartitioning are required; as the number of fields exchanged is high, the user decides to perform the interpolation on the extra interpolation processors (cIc). Between A and cIc, and between cIc and SI, repartitioning is automatically performed by the coupler communication library (c).
-O and SI have same grid and same distribution: the communication is direct (c to c).
 

- It was argued that A + O can not be run to any level of stationarity without SI if the region being modelled normally contains  any sea ice (since the feedback of ice cover to heat fluxes is important). A better example would probably be a system starting with A + O + SI. Then LS, AC and OB would be launched  when the simulation has reach some kind of equilibrium. This brings us back to the question if and which subsystems should be runable with PRISM.

Presentation of PALM

General presentation:

The PALM project aims to provide a general structure for a modular implementation of a data assimilation system. In this system, a data assimilation algorithm is split up into elementary "units" such as the observation operator, the computation of the correlation matrix of observational errors, the forecast model, etc. PALM ensures the synchronization of the units and drives the communication of the fields exchanged by the units and performs elementary algebra if required. This goal has to be achieved without a significant loss of performances if compared to a standard implementation. It is therefore necessary to design the PALM software in view of the following objectives and constraints:

• modularity: PALM provides a mechanism for synchronization of pre-defined functional units that can be executed in sequence, in concurrence, or in a mix of these two modes. One key aspect of PALM is also that dynamic or conditional executions of these units are allowed. PALM also performs the required exchange of information between these units.
• portability: PALM aims to run on all the existing high-performance platforms and, if possible, on the next generation super-computers. This effort of "clairvoyance" can be accomplished only through the adoption of standard.
• performances: PALM will be used in two modes: research and operational. The research mode will be used for the design of new algorithms and will prioritize the flexibility; on the contrary, the operational mode will work with a fixed configuration of the algorithm and will prioritize the performances optimization and the monitoring of the process.


More information can be found on the PALM WEB site.

A dynamic ocean-atmosphere coupled simulation with PALM:
 

A basic example of a dynamic ocean-atmosphere coupled simulation with PALM was presented. The example is based on a typical asynchronous coupled simulation. The ocean and atmosphere are run in a coupled mode for a fixed period of time in which the air-sea interface dynamics reaches some kind of equilibrium. The mean atmospheric fluxes are diagnosed near the end of this coupled period. The ocean is then run in a stand alone mode for a much longer period, forced by these diagnosed fluxes, in order to equilibrate the deep ocean dynamics. The whole cycle is repeated many times.

In the example, it is supposed that 7 processors are available. The simulation starts with an ocean model running with 4 processes coupled to an atmosphere model running with 2 processes;  this coupled configuration is run for a fixed period of 100 days. Between the ocean and the atmosphere, the coupling fields are transformed by an additional interpolator process, which also averages the atmospheric fluxes on the last 10 days of the coupled period. At the end of day 100, the ocean and the atmosphere are stopped. The ocean model, now running with 6 processes to take advantage of available resources, is re-started for 900 additional days in a stand-alone configuration. The whole cycle is repeated 10 times. This type of simulation, including loop structures for sequencing different configurations (coupled ocean-atmosphere or stand-alone ocean), and dynamic managing of the models (models are launched and stopped during the simulation), can be managed by PALM.
 

 
It was then argued that the dynamic launching and stopping of models is not required in this case.

The ocean could be started with 6 processes, the atmosphere with 2 processes and the interpolator with 1 process from the beginning; as only 7 processors are available, 4 processes (for example 2 ocean processes and 2 atmosphere processes) would have to run on the same 2 processors. During the coupled period, all model and interpolator processes would be active; during the ocean stand-alone period, the atmosphere and the interpolator would simply be waiting and possibly be swapped by the Operating System (OS) to avoid wasting of resources.

This simpler and more static configuration is attractive. It supposes however that:

• the computing centre policy and the batch system capabilities allow to run, for one particular job, more processes than the number of processors available.
• the memory is sufficient to run the 2 ocean processes and 2 atmosphere processes on 2 processors.

In a more general situation, for example with a more complete PRISM system integrating the other components, the total memory requested may be greater than the total memory available. But not all models may not be active at the same time; in that case, the proposed static configuration will requires that:

• an efficient OS swap functionality exists

Presentation of OASIS
 

The initial work on OASIS began in 1991 when the ``Climate Modelling and Global Change'' team at CERFACS was commissioned to build up a French Coupled Model from existing General Circulation Models (GCMs) developed independently by several laboratories (LODYC, CNRM, LMD). Quite clearly, the only way to answer these specifications was to create a very modular and flexible tool.

The main tasks of OASIS are:

the communication between the models being coupled, i.e. their synchronization and the exchange of the coupling fields at their interface

the transformation and interpolation of these coupling fields so that a target model can use the information given by a source model.

OASIS is a complete, self consistent and portable set of Fortran 77, Fortran 90 and C routines. It can run on any usual target for scientific computing (IBM RS6000 and SPs, SPARCs, SGIs, CRAY series, Fujitsu VPP series, NEC SX series, COMPAQ, etc.). OASIS can couple any number of models and exchange an arbitrary number of fields between these models at possibly different coupling frequencies. All the coupling parameters for OASIS (models, coupling fields, coupling frequencies, etc.) of the simulation are defined by the user in an input file namcouple read at run-time by OASIS.  Each component model of the coupled system remains a separate, possibly parallel, executable and is unchanged with respect to its own main options (like I/O or multitasking) compared to the uncoupled mode. However the models need to include few low-level coupling routines to deal with the export and import of coupling fields to/from OASIS.

To exchange the coupling fields between the models and the coupler in a synchronized way, four different types of communication are included in OASIS. In the PIPE technique, named CRAY pipes are used for synchronization of the models and the coupling fields are written and read in simple binary files. In the CLIM technique, the synchronization and the transfer of the coupling data are done by message passing based on PVM 3.3 or MPI2. In particular, this technique allows heterogeneous coupling. In the SIPC technique, using UNIX System V Inter Process Communication possibilities, the synchronization is ensured by semaphores and shared memory segments are used to exchange the coupling fields. The GMEM technique works similarly as the SIPC one but is based on the NEC global memory concept.

The fields given by one model to OASIS have to be processed and transformed so that they can be read and used directly by the receiving model. These transformations, or analyses, can be different for the different fields. First a pre-processing takes place which deals with rearranging the arrays according to OASIS convention, treating possible sea-land mismatch, and correcting the fields with external data if required. Then follows the interpolation of the fields required to go from one model grid to the other model grid. Many interpolation schemes are available: nearest neighbour, bilinear, bicubic, mesh averaging, gaussian. Additional transformations ensuring for example field conservation occur afterwards if required. Finally, the post processing puts the fields into the receiving model format.

More information can be found on the  OASIS WEB site.

Developments foreseen for OASIS within PRISM first year
 

The first deliverable of WP3a (D3a1) is described as "Portable PRISM coupler and model interface library, allowing a parallel communication between a parallel model and the sequential coupler, for preliminary interfacing of PRISM components" and should be ready at month 12 of the project. The developments planned for OASIS within PRISM first year to produce this deliverable are the following:

1. Model Technical Coupling Interface (MTCI)

The  Model Technical Coupling Interface (symbolized by I on the figures) is a set of high level routines that will encapsulate the low-level routines that the user has to implement in his model to deal  with the export and import of coupling fields. This MTCI will facilitate the user's work by:

• an automatic reading of some coupling parameters in the OASIS input file namcouple (this will in particular ensure coherence between OASIS and the models).
• allowing more flexibility in the models regarding the definition of other coupling parameters (e.g. coupling fields or exchange frequency) for a particular simulation.
• allowing a standard communication between the possibly parallel models and the present sequential OASIS coupler:

• allowing a parallel communication directly between two parallel models having same grid and same decomposition, in which case neither interpolation nor redistribution are needed:

First suggestions for the Model Technical Coupling Interface are presented in 6.

2. Interfacing with the SCRIP interpolation library

The SCRIP (Spherical Co-ordinate Remapping and Interpolation Package) sequential library, developed by the Los Alamos National Laboratory, USA, will be interfaced in OASIS. Compared to OASIS today, the added 2D interpolation functionalities will be:

• Bilinear and bicubic interpolation schemes for all grids labelled with logically rectangular indices, e.g. (i,j) (OASIS bilinear and bicubic schemes today are not adapted if the source grid is not a lat-lon grid).
• A first and second order conservative remapping adapted for any grid which meshes can be described by an arbitrary number of corners (OASIS SURFMESH conservative remapping is only adapted for lat-lon grids, either as source or target grid; in OASIS MOZAIC operation, the remapping set of weights and addresses must be provided by the user).

A survey of other existing interpolation libraries should also be undertaken, as well as an interaction with the authors of the SCRIP library to learn about their plans for parallelization of the library.

3. Pseudo parallelization of OASIS on a field-per-field basis

Different approaches of parallelization for OASIS have been discussed. The full parallelization of OASIS via data partitioning which implies also the parallelization of the interpolation library will be achieved for the final version of the PRISM coupler but is not foreseen within the first year. One advantage of this full parallelization will be that the number of MPI messages between application and coupler is reduced compared to approach b.i. provided that all fields of each process are sent or received with one MPI call. This will become of increasing importance when coupling massive parallel applications.

As OASIS usually treats and exchanges many coupling fields, a pseudo parallelization of OASIS, within PRISM first year, on a field-per-field basis was proposed. OASIS would run with different processes, each one receiving, interpolating and sending one or more entire coupling fields. If OASIS is used to exchange many fields, this could prevent OASIS to become a bottleneck in the simulation. Furthermore, as in OASIS many coupling fields cannot be packed into one message -each coupling field is necessarily a message on its own-, this would not cause any overhead in communication compared to the usual sequential OASIS. This first step would be easy to achieve as the interpolation library itself would not need to be parallelized.
 

Different approaches of pseudo-parallelization for OASIS were discussed.  A decision has not been made. Following solutions are possible:

• Bidirectional approach:

each OASIS process  treats an equal number of fields exchanged in each direction. For example, if 8 fields are exchanged from A to O and 4 from O to A, and if OASIS is run with 2 processes, then each OASIS process treats 4 fields from A to  O and 2 from O to A.
• Pros: this approach ensures a good load balance for the OASIS processes.
• Cons: each OASIS process has to calculate the information required for the transformation, e.g. the interpolation matrix of weights and addresses.
• Unidirectional approach:

one OASIS process treats all coupling fields exchanged in one direction, e.g. 8 from A to O, and another OASIS process treats all coupling fields exchanged in the other direction, e.g. 4 from O to A.
• Pros: each process has to calculate only the information to do the transformation in one direction. This is no longer an advantage when the transformation information in one direction can be automatically deduced from the transformation information in the other direction.
• Cons: the load balance between OASIS processes is no longer ensured.
• Component approach:

if OASIS is coupling more than two models, another possibility is to have one process between any two models.
• Pros: the transformation information is optimally calculated by the different OASIS processes and not duplicated.
• Cons: the load balance between OASIS processes is no longer ensured.

It was also argued that the second option may not bring any benefit if the coupling in one direction occurs at a different time than the coupling in the other direction. The same remark applies for the third option if the coupling between any two models occurs at a different time than the coupling between any two models. In these cases, a sequential coupler would always be available to receive, treat and send the coupling fields as soon as they become  available. This argument does not necessarily hold for OASIS today: at each coupling timestep, OASIS first receives all coupling fields exchanged at a particular time, then transforms them all, then sends them all. An easy optimization would be to change the order of the internal OASIS loops so that each coupling field is transformed and sent out as soon as it is received.

It was therefore decided that this last optimization should first be implemented if possible in OASIS. Based on the pros and cons listed above, it should be evaluated if any of the pseudo parallelization approach would then bring some benefits for OASIS.

4. Optional use of MPI1 in CLIM

In the CLIM/MPI2 communication technique in OASIS, the only MPI2 feature is the spawning of the models by OASIS at the beginning of the simulation (MPI_COMM_SPAWN). It was concluded to keep the MPI2 feature in the CLIM communication technique, but to add an MPI1 option, without MPI_COMM_SPAWN, for those platforms which do not support MPI2 but do have an MPMD implementation of MPI1. In CLIM/MPI1, all codes started at the beginning of the simulation automatically will share the same MPI_COMM_WORLD communicator; therefore there will be a need in the initialization phase to perform a MPI_COMM_SPLIT and generate for each model a local communicator for internal parallelization.

5. Other miscellaneous improvements

Other miscellaneous improvements will include:

• partial rewriting in Fortran 90 to take advantage of F90 dynamic memory allocation (today a pseudo dynamic allocation is set up by defining a list of macro arrays which sizes are defined through parameter statements for handling the input and output coupling fields, the input and output grid related parameters, etc.)
• support of netCDF format for all OASIS restart and transformation auxiliary files
• simplification of the namcouple input file

First suggestions for the  Model Technical Coupling Interface (MTCI)
 

First suggestions for the  Model Technical Coupling Interface (MTCI) were then presented. An exhaustive list of possible requirements for MTCI can be found here. First ideas about a concrete MTCI to the PRISM coupler are given here.
In addition to the general requirements, the following technical points were discussed in more detail:

• The proposed programming language for the PRISM coupler is Fortran 90 and C to ensure and facilitate portability to a large variety of platforms.
• The good trade-off has to be found between (I) a concise list of parameters for each subroutine call (more subroutines provided with a shorter list of parameters) and (II) a small number of subroutines (but a longer and more complex parameter list). The complexity arises from the need to transfer not only the coupling data but also the meta-data, i.e. the description of the data (e.g. units, grid co-ordinates, mask, length of a time step, distribution, ...). Providing the user with a stable interface which at the same time allows the developers to extend functionality on demand can best be guaranteed with option I: for each type of data a separate subroutine is provided for defining the meta-data as well as sending and receiving these data . Option II would probably lead to less efficient subroutines.
• For performance reasons at this stage it was preferred to define handles to the grid functions as integer numbers instead of using character strings that have to be communicated to the coupler through the MTCI. One suggestion was to provide a PRISM include file that contains a parameter list of integer numbers to identify the physical variables following e.g. the Grib convention. In this PRISM include file all physical variables that are send and/or required by any application code have to be considered.

The transformer part of the coupler has to understand the description the data, i.e. meta-data, that will be transfered from the model to the coupler through the MTCI. The PRISM standard meta-data will be defined in WP2c. It was therefore suggested that a meeting should be arranged between the WP3a, WP2c and WP4b groups to make sure that we participate to this definition and to evaluate how far the model interfaces for coupling data and other I/O data should be harmonized.

Distribution of WP3a tasks

The distribution of the WP3a tasks among the partners was then discussed. Everyone agreed on the following proposition. It is based in particular on 24 and 12 pers. months for respectively NEC-CCRLE and NEC-ESS, and not on 48 and 24 as it appears in Annex 1 of the contract; we suppose that these numbers of 24 and 12 pers. months will be officially acknowledged at the kick-off meeting.
 

Contributing partner	CERFACS #8	NEC-  CCRLE #22	NEC-  ESS #18	Fecit #19	SGI #20	MPI-M&D #3	Total
Pers. months	38	24	12	6	4	2	86
-----------	-----------	-----------	-----------	-----------	-----------	-----------	-----------
T1-Specifications	2	2	2	1	2	0	9
T2a- OASIS 2.5	6	0	1	1	0	1	9
T2b- Model Gen. Interface	3	1	0	1	0	0	5
T3- PRISM coupler driver	9	0	1	1	0	0	11
T4- PRISM coupler interpolation library	11	0	4	0	1	0	16
T5- PRISM coupler communication library	2	19	0	1	0	0	22
T6- Test prototype version of coupler	2	1	3	1	1	1	9
T7-PRISM coupler  documentation	3	1	1	0	0	0	5
-----------	-----------	-----------	-----------	-----------	-----------	-----------	-----------
Total	38	24	12	6	4	2	86