Participants: Rene Redler and Hubert Ritzdorf (NEC-CCRL), Reiner Vogelsang (SGI Deutschland), Jan Polcher and Philippe Bourcier (IPSL), Sophie Valcke (CERFACS).
Location: salle retonde IPSL - Tower 16, 4th floor, IPSL, Université Pierre et Marie Curie, 4 place Jussieu, F-75252 PARIS.

The points raised during the meeting for which an agreement was found or which still require further discussion are described here below.

• Coherence checks:
• One piece of information, given by the argument "name", will be used to identify the variable or the grid in the SMIOC/SCC (cf routine prism_def_var for transient variables, or routine prism_def_grid for grids). This character string "name" argument can be composed of any character, can be of any length, and will not be case sensitive.
• "SMIOC_name" will not appear as an argument for the prism_init_comp routine. The name of the model "model_name" will be used to construct automatically  the SMIOC name; this character string "model_name" argument can be composed of any character, can be of any length, and will not be case sensitive.
• All metadata present in the SMIOC should be accessible in the model code by calling the prism_get_persist routine. For example, the model version number could be accessed for a coherence check implemented by the developer in the model code if desired. (Is a comp_id needed in the prism_get_persist call?)
• No prism_def_var_byid routine will be developed.
• Levels of variables transferred:
• The set of "minimal operations", required to project the model representation of the transient variable data onto the grid specified in the grid declaration calls, will be listed in the PMIOD by the developer and reported automatically in the SMIOC. The user will not be allowed to change this set of "minimal operation" that will be performed automatically by the PSMILe below the prism_put/prism_get. The variable "units" indicated in the PMIOD/SMIOC will be the units after the minimal operations if any.
• The scattering or gathering "minimal operations" can use a simple mask declared by a prism_set_mask and identified by an ID mask_id in the prism_def_var call. The other "minimal operations" are local point-wise operations. Therefore, no [Y] array is needed in the prism_put/prism_get calls. The mask will, of course, have to be updated through the PSMILe interface if it changes in the model code.
• Grid declaration:
• Comparison with FMS:

Contrary to FMS approach, the grid declaration and definition will be done through the PSMILe interface and not in separate files
• The local grid and variable declaration will be done locally to each model process with the following calls:
1- the grid definition:
=> prism_def_grid (grid_name, grid_id, comp_id, grid_valid_shape, grid_nodims, grid_type, ierror )
• grid_name [IN]: the grid name for identification in the SMIOC
• grid_id [OUT]: a returned ID
• comp_id [IN]: the component ID
• grid_valid_shape: ilow,ihigh,[jlow,jhigh],[klow,khigh] of valid range, without halo regions.
• grid_nodims: the rank of the arrays describing the grid (e.g. grid_valid_shape)
• grid_type: PRISM integer named parameter describing the structure of the grid. The exact list of grid_type still has to be worked out. For the vertical dimension the letters "vrt" were proposed instead of "dep". It has been agreed that all grids should be declared as 3D grids with possibly degenerated dimensions of extent 1 (e.g. a surface 2D grid would be expressed as a 3D grid with a 3rd dimension of extent 1).

2- the grid corner (control volume) declaration:

=> prism_set_corner (grid_id, nbr_corners, corner_actual_shape, 1stcoord_array, 2ndcoord_array, 3rdcoord_array, ierror)
• grid_id [IN]: the grid ID
• nbr_corners: the number of corners. Should be a vector giving the number of corners in each physical dimension?
• corner_actual_shape [IN]: ilow,ihigh,[jlow,jhigh],[klow,khigh] of actual region; must cover at least grid_valid_shape region but can also include halo regions.
• 1stcoord_array [IN]: the longitude of the grid corners (the shape this array depends on  grid_type)
• 2ndcoord_array [IN]: the longitude of the grid corners (the shape this array depends on  grid_type)
• 3rdcoord_array [IN]: the vertical position of the grid corners (the shape this array depends on  grid_type)
Note 1: The valid shape of the corner arrays are necessarily the same as the valid shape for the grid, grid_valid_shape.
Note 2: It remains to be clarified if the prism_set_corner call is mandatory.
Note 3: The number of corners could be determined automatically based on the grid_type. We may want to keep it here to have more flexibility (e.g. to define 8 corners per 2D cell).
 
3- the positioning of points in the control volume:
=> prism_set_point (grid_id, point_id, 'method', center_actual_shape, 1stcoord_array, 2ndcoord_array, 3rdcoord_array, ierror )
• grid_id [IN]: the grid ID
• point_id [OUT]: the returned point ID; will be used in the variable declaration call 'prism_def_var' to link the point definition to the variable.
• 'method': a PRISM integer named parameter describing the position of the the point in the control volume (e.g. CENTER, NORTH, EAST, NORTHEAST,... ; other 'methods' can be defined for other type -e.g. triangular- of grids); for each variable, its 'method' will be indicated in the SMIOC (therefore, maybe not needed here?).
• point_actual_shape [IN]: ilow,ihigh,[jlow,jhigh],[klow,khigh] of actual region; must cover at least grid_valid_shape region but can also include halo regions.
• 1stcoord_array [IN]: the longitude of the grid points (the shape this array depends on  grid_type)
• 2ndcoord_array [IN]: the longitude of the grid points (the shape this array depends on  grid_type)
• 3rdcoord_array [IN]: the vertical position of the grid points (the shape this array depends on  grid_type)

Note: this routine can be called many times for the same grid_id with different 'methods' to place different points in the grid control volume.

4- possibly, the subgrid fractions:

=> prism_set_subgrid (grid_id, sub_id, 'FRACTION', nbr_types, fraction_actual_shape, fraction_array, ierror)
• grid_id [IN]: the grid ID
• sub_id [OUT]: the returned subgrid ID; will be used in the variable declaration call 'prism_def_var' to link the subgrid definition to the subgrid variable (e.g fluxes for x vegetation types).
• 'FRACTION': a PRISM integer named parameter describing the subgrid method; for each variable, its 'method' will be indicated in the SMIOC. (It has to be clarified however if there are other subgrid methods than FRACTION; if not, this argument is not be needed.)
• fraction_actual_shape [IN]: ilow,ihigh,[jlow,jhigh],[klow,khigh], nlow, nhigh of actual region; ilow,ihigh,[jlow,jhigh],[klow,khigh] must cover at least grid_valid_shape region but can also include halo regions; nlow,nhigh give the number of subgrid types.
• fraction_array [IN]: the fraction for each type (the shape this array depends on  grid_type)

5- possibly, the definition of other characteristics (the areas or the scalefactors, the masks -possibly more than one mask per grid-, the local rotation angle, the connectivity ?, etc.)


6- the variable declaration:

=> prism_def_var (var_id, name, point_id or sub_id, mask_id, comp_id, var_nodims, var_actual_shape, var_type, ierror)
• var_id [OUT]: the returned variable ID; to be used in the prism_put/prism_get
• name [IN]: the variable name for identification in the SMIOC/SCC
• point_id or sub_id [IN]: the point  ID or the subgrid ID
• mask_id [IN]: the mask ID
• comp_di [IN]; the component ID
• var_nodims [IN]; var_nodims(1): the overall number of dimensions of array; var_nodims(2): number of bundles, if any (0 if not a bundle).
• Units:

Units for grids, transient and persistent variables will be indicated in the PMIOD/SMIOC, not in the grid or variable declaration calls.

• For grids, the only units supported in a first step will be degrees_east and degrees_north (spherical coordinate system). The Driver could check if these defaults units are indicated in the PMIOD/SMIOC. Other units will eventually be supported later when appropriate automatic conversion will be implemented.
• For transient variable units, the user will therefore be able to check if the units of an exchanged variable match and if not, will have to indicate appropriate transformations. A check of coherence by the Driver or by the User Interface will also be possible based on the PMIOD/SMIOC information.
• For local persistent variable, the units indicated in the PMIOD/SMIOC are the ones used by the model. For global/universal persistent variables (also called universal parameters), defaults units should also be agreed on. The Driver could check if these defaults units are indicated in the PMIOD/SMIOC/SCC. Other units than the default ones will eventually be supported later when appropriate automatic conversion will be implemented.
• Implication of (vertical) grid evolving during the simulation:
• Even if the horizontal and vertical grid definitions are not separated in the initial declaration call, a way to indicate at run-time which part of the grid has changed and to update only the part of the grid that changed is needed. This is required as in many cases only the vertical definition of the grid will change and the whole grid definition should not be transferred through the PSMIle each time. An additional routine prism_update was suggested, but the exact form of this PSMILe functionality still has to be worked out (e.g. should there be a particular grid_type announcing that the vertical definition of the grid will change during the run?)
• Implication of other grid characteristics (mask, areas, subgrid fraction) evolving during the run:
• All grid characteristics (mask, areas, subgrid fraction) will be allowed to evolve during the run. The exact form of this PSMILe functionality still has to be worked out.
• Other topics not discussed in detail:
• How to transfer the vertical definition of the grid through the PSMILe interface?
• by transfering the Z levels each time they change?
• by "pointing" to a transient variable which will be transferred through the inteface with  prism_put calls?
• could only the pressure or isopycnal levels (no Z levels) be transferred intially through the interface? Is that sufficient for interpolation needs? For I/O needs?
• Repartitioning method  (discussed on the 19/09)
• Default units for vertical grids?

19/09 WP3a-WP4a PSMILe development meeting

Participants: Rene Redler and Hubert Ritzdorf (NEC-CCRL); Reiner Vogelsang (SGI Deutschland); Thomas Schoenemeyer (NEC Deutschland); Jan Polcher, Marie-Alice Foujols, Philippe Bourcier and Alexandre Tallec(IPSL); Stephanie Legutke (MPI/M&D), Jean Latour (FECIT/Fujitsu), Sophie Valcke and Damien Declat(CERFACS).
Location: IPSL meeting room, in between Tower 24 et 25, basement level, IPSL, Université Pierre et Marie Curie, 4 place Jussieu, F-75252 PARIS.

The points raised during the meeting for which an agreement was found or which still require further discussion are described here below.

• PSMILe internal element structure and responsibilities of partners in the PSMILe development:

The following PSMILe internal structure for the PSMILe was proposed and discussed:

• Everyone agreed that it is possible and desirable to build modules with well-defined interface for the Local transformation, Calendar, Access to SMIOC and SCC, Input/Output, Error handling and Monitoring boxes.
• The main point of discussion was on the possibility to achieve such a modular structure, with precise API between the modules, for the Execution tree, Data Management, Metadata management, Internal Interface to I/O and coupling and Coupling boxes, and on how to test separately these different boxes.
• The content of the Metadata management module also needs to be rediscussed and we need to agree on whether or not all the grid information (longitudes, latitudes, vertical levels, masks, areas, etc.) is part of the Metadate management module or if this module contains only the Metadata read from the SMIOC/SCC.
• It was agreed that Philippe, Reiner, Hubert and Rene should meet to solve these two issues and present a preferred solution to the group at our next meeting.
• In previous meetings, it was agreed that, for not stand-alone models, the SCC information should be read by the Driver and transferred to the model PSMILe by message passing. It was also proposed at this meeting that the Driver could play the same role for the SMIOC information, as the disk on which the SMIOC file resides may not be easily accessible to all model processes at run-time. It has to be clarified how the XML tree information could be transferred from the Driver to the model PSMILes.
• An agreement was found on the following list of responsibilities for each partner in the PSMILe development. The responsibility means that the partner is in charge of supervising and managing the development of a particular part of the PSMILe, not that the partner has to develop that whole part alone. Some responsibilities may be shared by two partners.

• From the module perspective:
• NEC-CCRLE:
• Execution tree
• Internal interface to (I/O) and coupling
• Data Management
• Metadata management
• Coupling
• IPSL:
• Calendar
• Access to SMIOC and SCC
• Internal interface to I/O (and coupling)
• Metadata management
• Input/output
• Error handling
• Monitoring
• CERFACS
• Local transformations
SGI Deutschland will be involved in the development of the Input/Output module. CERFACS is also responsible for the development of the other Coupler entities, i.e. the Driver and the Transformer.
 
• From the PSMILe interface routine perspective:
• NEC-CCRLE:
• 2 - Routines for initialisation, setup, and termination: PRISM_init , PRISM_init_comp, PRISM_get_localcomm, PRISM_initialized (?),  PRISM_setup/enddef,  PRISM_decommission(?), PRISM_terminate, PRISM_terminated (?)
• 3 - Routines for grid declaration: PRISM_def_grid, PRISM_set_center_xx, PRISM_set_xcorners_xx, PRISM_set_mask, PRISM_set_scalefactor, PRISM_set_angle, ...
• 4 - Routines for transient variable declaration: PRISM_def_var
• 5 - Routines for sending and receiving transient variables: PRISM_put, PRISM_get, PRISM_iput,  PRISM_iget, PRISM_wait
• 6 - Different utility routines for the component models: PRISM_query_var, PRISM_error
• IPSL:
• 4 - Routines for transient variable declaration: PRISM_def_restvar
• 6 - Different utility routines for the component models: PRISM_get_persist, PRISM_def_startdate, PRISM_calc_newdate, PRISM_calc_nbtstp,  PRISM_error
• CERFACS:
• 3 - Routines for grid declaration: PRISM_def_grid, PRISM_set_point_xx, PRISM_set_corners_xx, PRISM_set_mask, PRISM_set_scalefactor, PRISM_set_angle, ...
• 6 - Different utility routines for the component models: PRISM_timestamp, PRISM_restartsaved, PRISM_message.
• Coding rules:
• It was agreed that the GNU tools proposed by WP2b should be used for the Coupler (PSMILe + Driver + Transformer) development.
• Even if none were proposed by WP2b, it was agreed that the development and the maintenance of the software would greatly benefit from the adoption of the  same coding standard (e.g. routines and variable names) by all WP3a/WP4a developers. Sophie will propose such a norm shortly.
• The need of a central CVS server is now obvious. This request will be forwared to the PRISM Director. Meanwhile, the partners will keep their own developments on their local CVS server.
• PRISM Coupler V.0 (Oasis 3.0 + PSMILe V.0):

Version 0 of the PRISM Coupler, composed in fact of Oasis 3.0 and PSMILe V.0 will be delivered in December 2002. Compared to Oasis 2.4, the main improvements are:

• F90 memory dynamic allocation (completed)
• Namcouple simplifications (completed)
• NetCDF format for grid and restart auxiliary files (completed)
• Interfacing with SCRIP interpolation library (in progress- Sophie)
• PSMILe V.0:
• Adaptation of CLIM/MPI1 and CLIM/MPI2 interface to the current definition of the (final) PSMILe interface (still to do: date format in the prism_put/prism_get; names of routines with suffix "_proto" )
• Direct communication between two models with same grid and same distribution (completed)
• Automatic writing of (NetCDF) restart below PRISM_put at the end of the simulation in sequential (completed)
• Automatic writing of  (NetCDF) restart below PRISM_put at the end of the simulation in parallel (in progress - Arnaud)
• Transfer of INSTANTaneaous, AVERAGEd, or ACCUMULated field below the PRISM_put (to do - Jan + Marie-Alice + Arnaud)
• Adaptation of the PRISM_put and PRISM_get `date' argument so that it follows the final PRISM date format (to do - Jan + Marie-Alice + Arnaud)
• Inclusion of IOIPSL and/or mpp_IO basic I/O routines below PRISM_put and PRISM_get (to do - Jan + Marie-Alice + Arnaud)
• Use of GNU tools for code development (automake, configure, ...) (to do - Marie-Alice + Arnaud)
• Improvement of the  toymodels so that it makes a complete test of PSMILe V.0 functionalities (to do - Marie-Alice + Arnaud)
• Presentation on FMS mpp_io and discussion on repartitioning(stitching) method

The GFDL Flexible Model System I/O library, mpp_io, written by Balaji, was then presented by Reiner. The main conclusions of this presentation, than can be found at http://www.cerfacs.fr/PRISM/MTCI/Reiner_pres_190902/img0.html are the following:

• For parallel model outputs, two options are offered: either one file per process is written, either the entire field is gathered on the master process which writes the file. After preliminary tests, it appears that the second option needs the description of each local partition in the global index space to reconstruct the global field. (Note on the 27/09: Reiner confirmed that the global index space is needed for the second option).
• The problem is that this description of the local partitions in the global index space may not exist for some models (e.g. GME), simply because the global index space concept is not relevant for these models.
• It was therefore decided to base the PSMILe coupling repartitioning algorithm on the description of the local partition in the physical space (longitude and latitudes).
• However, the description of the local partitions in the global index space, when available, could still be transferred through the PSMILe interface with a separate routine. The exact form of this PSMILe functionality still has to be worked out by Marie-Alice, Reiner and Sophie.This decision solves a problem for the PRISM Coupler V.0 as Oasis needs this local description in the global index space for parallel exchange to/from a parallel model. When it exists, the local description in the global index space could possibly be used to optimize neighborhood search.
• One possibility for each particular parallel model outputs /inputs is to build a sequential pseudo-model having the equivalent global grid. The PSMILe coupling library could then be used to do the repartioning between the parallel model and the respective sequential pseudo-model (or vice-versa) which would then perform the outputs (or the inputs). This is analog to what was proposed earlier by G. Berti (see "A suggestion for the coupler on how to perform parallel I/O " at http://www.cerfacs.fr/PRISM/MTCI/coupler-and-io_GB_240102.ps)
• PMIOD DTD structure:

Alexandre then presented few slides on the DTD structure for the PMIOD xml file.

• One conclusion is that the PMIOD DTD structure has to be reviewed to include the 18-19/09 meeting decisions, and should be finalized by mid-November.
• More work will then be required to establish the SMIOC and SCC DTD structure. It would be usefull to have a prototype version of these DTD by the end of the year.
• Another important conclusion is that interaction is needed between Alexandre, Philippe and Claes Larsson (WP4a) as the tools to build the SMIOC and the SCC in a coherent way given the different PMIODs should be developed as part of the User Interface.
• Access to SMIOC and SCC (libxml):

Philippe then made a presentation on Access to SMIOC and SCC (libxml) (http://www.ipsl.jussieu.fr/~pbipsl/prism/doc/SMIOCandSCCaccess/frames.html) .

• Driver and Transformer flow chart:

Damien then presented a suggestion for the flow Driver and Transformer flow charts. His presentation can be found on http://www.cerfacs.fr/PRISM/INTERPOLATION/WP3a_transformer_Sep02.ppt
 

• Flow chart for PSMILe initialisation, setup, and termination routines  and for PSMILe sending and receiving transient variables routines:

Rene then presented a first suggestion for the flow charts of different PSMILe routines. This first suggestion needs to be updated according to Damien's suggestion for the Transformer flow chart.
 

• Separate working sessions:

The group then splitted into two working sessions.

• Session I was on PSMILe, Driver, Transformer interaction. Some conclusions are:
• Programming the Transformer and PSMILe are separate tasks.
• It still has to solved where the Driver is located (separate process of part of the Transformer).
• The way how to pass the contents of the XML tree from the Driver to the applications needs to be clarified - still an open issue.
• Where to do the neighborhood search for the interpolation (in the Transformer or in the PSMILe) is not solved.
• Session II was on the PSMILe Utility routines.
• The prism_put/prism_get date format was discussed. It was agreed that the date should be expressed as a vector of 6 integers (year, month, day,hour,minute, second).
• It was also agreed on that Marie-Alice, Jan and Arnaud are responsible for adapting the PSMILe V.0 prism_put and prism_get routines so that the date argument expressed as a vector of 6 integers is transformed internally in  "number-of-seconds-since-the-beginning-of-the-simulation", as this last format is the one needed by Oasis.