WP3a meeting
KNMI, October 6th, 2004

Minutes (draft, 07/10/2004)

 

 

• Location: KNMI, DeBilt. Room: Vening Meineszzaal

• Time: 10h00 - 18h00

• Participants:
• René Redler, Hubert Ritzdorf (NEC-CCRLE) 
• Reiner Vogelsang (SGI Germany)
• Thomas Schoenemeyer (NEC)
• Stephanie Legutke (MPI, M&D)
• Sophie Valcke (CERFACS)

The following points were discussed:

• Info on ESMF-PRISM interaction (S. Valcke) :

Sophie reported on the ESMF-PRISM meeting that took place in Princeton in October. The PRISM and ESMF projects were first presented. ESMF aims at defining a coding  standard with corresponding standard coding tools (e.g. standard ESMF structures for grid, field, and component declaration, standard routines for internal parallelisation, for communication with other parts of the code, etc.). This should help users to build a code, possibly composed of a hierarchy of components. One aspect not addressed in ESMF yet is the communication between different executables, whereas this is one main focus of PRISM. The PRISM and ESMF developers agreed to interact on the following points:

• Calendar tool: PRISM will evaluate ESMF calendar tool; it could be used internally in OASIS4 and it could be proposed to the model developers as a standard calendar tools to be used in the models.

• ESMF agreed to test the use of OASIS4 as coupler between two ESMF codes. The first test case should be MOM4 (using ESMF standards) coupled with a toy atmosphere model. A proposal should be submitted to NASA to get fundings for this. CCRLE, SGI and CERFACS will be identified as “collaborators”, which should get them some limited amount of money to interact with ESMF people and travel to the USA.

• XML description and configuration: PRISM and ESMF agreed to interact on XML component code description and on XML coupled model configuration. This interaction will be based on the AD, PMIOD, SCC, SMIOC XML files presently used in OASIS4. Katherine Bouton from BADC is also working on defining a standard XML DTD for full code description (not only the coupling interface as we do it for now in the AD and PMIOD). PRISM and ESMF also agreed to interact on a standard grid XML description.

• Report on MOM4 and OPA9 interfacing with OASIS4 (R. Redler)

The coupling between MOM4 and a toy atmosphere model is technically working. This was done with minimal interference in the codes (routines using PSMILe are all present into one separated module). Some improvements had to be brought to the mpp package used internally by MOM4 to allow use of an internal communicator instead of  MPI_COMM_WORLD in the coupled case. Direct communication as well as communication through the Transformer with trilinear interpolation is tested. Component models are not parallel for now. Complexity will be progressively added (parallel models, parallel Transformer, repartitioning, other interpolations, etc.)

René is also interacting with Mojib Latif’s group in Kiel regarding OPA9 interfacing with OASIS4. Presently, this interfacing is done but not tested. OPA9 is the OPA version used in the French DRAKKAR project. In Kiel, they are presently using the OPA8 coupled to ECHAM5 with OASIS3, but they might want evolve toward OASIS4 at a later stage.

• Review of current developments and priorities for OASIS4 final version (see reviewed html document)
• SV makes sure that mpp_io versions used in OASIS3 and OASIS4 are identical.
• Problems with PGF90 compilation : SV checks if there are still some problems with OASIS4 PGF90 compilation (mpp_io, time required)
• Problems with SGI compilation (elnino): SV checks what is the memory limit on elnino (-hinv) and reports to RV. RV traces memory required by the different parts of OASIS4.
• Test of ESMF calendar tool (RR): For now, we wait for ESMF improvement to support proleptic Gregorian calendar before -4800.
• Clean up prism.inc (ECMWF local codes)
• Real and Double precision declaration vs PRISM coding rules?  We decided not to follow the PRISM coding rules because the PSMILe uses generic interface routines for all API routines. If we add the interface for Quadruple and Douple Quadruple types, we would cover all possible type declarations in the applications (we would therefore be even more flexible than suggested in the Coding Rules).
• Lag and restarts fully implemented. RR modifies PSMILe behaviour so to abort if lag is defined and no restart file is present.
• Multigrid algorithm description (in 3D):
1. The 8-corner bounding boxe of the local geographical domain (rectangular box in the coordinate units) is computed by each process PSMILe.
2. The bouding boxes of all processes sent to and collected by all processes.
3. Each source process calculates the intersection of its bounding box with each other process bounding box, thereby identifying the potential interpolation partners and corresponding bounding box intersection.
4. A multigrid sequence is applied by each source process on each partner bounding box intersection:

• 1^st level: The 8-corner bounding boxes and mid-point for each source cell in the intersection is calculated (those 8-corner bounding boxes might overlap).
• 2^nd level corresponds to the calculation of the 8-corner bounding boxes for the 1^st level 8-corner bounding boxes coarsened by a factor of 2
• 3^rd level corresponds to the calculation of the 8-corner bounding boxes of the 2^nd level 8-corner bounding boxes coarsened by a factor of 2.
• … and so on until falling back onto the 8-corner bounding box covering the whole intersection.

5. For each target grid point in each partner intersection, it is then searched into which source bounding box the target point falls for each level of the multigrid sequence, starting by the coarser level. If the target point falls into 2 source bounding boxes, the one for which the distance to the mid-point is smaller is chosen. If no bounding box is found at the next (finer) level, then the search goes back to the previous (coarser) level. Once it is found into which finer source bounding box the target point falls, it is searched into which “method cell” the point falls. The “method cell” is the cell formed by 8 grid points (as opposed to corner points). For linear interpolation algorithm, the search is finished and those 8 method cell grid points will be transferred to the Transformer. For nearest-neighbour interpolation algorithm, the search consider also the points into the (10, 18, 24 ????) direct neighbour cells  of the method cell . If a coupling involves different fields and different interpolation techniques, the results of one multigrid algorithm search can be re-used for another interpolation technique or another coupling field.

• SV checks if definition of element  if_masked is in the smioc.dtd checked in driver/xmlfiles and if transfer to PSMILe is done correctly.

• If a target point falls outside any source method cell, no interpolation is calculated  (is this true for tri-bi-linear algorithm only or also for nneighbours also?). Sophie will define an additional element in the smioc so that the user can choose the PSMILe  behaviour in that case (no interpolation, nearest-neighbour value, etc.).

• 2D1D interpolation combination that work in Ps are: bilinear and linear, bilinear and none, nearest-neighbour and none. SV has to review those combinations in T. We could easily implement other 2D1D combinations but we then have to define new interface for Ps-T transfer (e.g. if 2D1D with bilinear and 8-nearest-neighbour in the vertical is requested).

• René will provide a test case for a coupling involving an application having more than one component running sequentially to test that functionality.

• Reiner will implement the use of  fill_value with a tolerance so that even if there is some type conversion of data going through the T the data points to replace by a fill_value (or vice-versa) will be detected.

• Sophie will install a CVS server at CERFACS for OASIS4 day-to-day developments. On bedano, only tagged versions will be checked-in.

• Scalability tests:

o       Reiner tags OASIS4 version after modifications about fill_value and latest Hubert’s changes (end of next week).

o       René instruments simple-mg example to measure:

§         the total wall clock time

§         the time used for prism_enddef

§         the time between beginning prism_put and end of prism_get on the same side (ping-pong).

o       René sends compiling flags with which the tests have to be preformed.

o       Starting configuration for the simple_mg example models is T106 for appl-atm and 1/3X1/3 for appl-ocn. The following numbers of processes will be tested for respectively appl-atm, Transformer, appl-ocn: 1-1-1, 2-1-2, 4-1-4, 8-1-8, 2-2-2, 4-2-4, 8-2-8, 4-4-4, 8-4-8, 8-8-8. René performs tests on Linux PC cluster, Thomas on NES SX6, Reiner on IRIX MIPS and Altix 3000, Jean on Opteron, Sophie possibly on IBM. Deadline for the tests: end of October. Thomas gathers the results and writes a first draft report for mid-November.

• Documentation: Sophie revises her first draft, checks-in the LaTex files on OASIS4 CERFACS CVS server, and sends around a revised draft for end of October. Reiner will review the part “Getting the sources”, René the part “PSMILE API” and the part “Compiling and Running”,  and Thomas the “Driver” and the “Transformer” parts. Sophie will build the corresponding html tree and will give the html link so that people accessing the previous PSMILe API web site automatically fall onto the updated documentation. The full dtd should not appear in the pdf version (only the html link should appear. René will also build the full ProTex documentation so that a link can be made from the PRISM Web site.

• Sophie will adapt OASIS4 to PRISM standard SCE and SRE but only after the official EU release.

• Hubert has started programming the exact parallel neighbour search but this will not be included in the official EU release.