minutes_wp3a

Oasis4 follow-on developments
(13/06/2005)

In blue, work done by RR, HR
In purple, work done by DD, SV
In green, work done by RV
In dark green, work done by TS

High priority developments

Install CVS server at CERFACS for developments (SV).
Documentation; scalability tests (SV & all; see minutes of 06/10/2004 meeting)
Report on correctness of interpolation (SV)
Make sure that mpp_io versions used in OASIS3 and OASIS4 are identical (SV).
Debug proto_ex example. Why is transi_out_id sent by ocean PS to T is always 2? Determination of global IDs by Driver seems OK in all cases. (RR, HR, DD)

Treatment of lag and associated automatic writing and reading of coupling restart (cold and normal restarts) (RR)

lag is defined in terms of "prism_put period", i.e. the difference between the associated date_bounds. E.g. for a lag of 1, the time to add to the prism_put date and date_bounds arguments would be 1 X the difference between the associated date_bounds. There is still the problem that for models with varying timestep, the "lagged" period calculated that way may not exactly cover the period that would have been covered by the next prism_put, as the timestep may vary. (We could add an additional argument giving explicitely the period covered by the next prism_put, but even then this information may not be known by the model.)
below the last prism_put, fields with lag should be written to files (one file per process with field name, component name, application name, process number and run final date)
at restart, at the end of prism_enddef, fields with lag will be read by PSMILe and send to Transformer.
Implement abort if lag is defined and no restart present (René)

Interpolation:

3D nearest-neighbour (nneighbour3D) and trilinear ( trilinear)

PS: nneighbour3D: use of user's choice for nbr_neighbours value (HR)
T: nneighbour3D: use of user's choice for gaussian_variance (DD)
PS: nneighbour3D: use of user's choice for used_masked value (used_masked=false or true) (HR)
PS: trilinear: use of user's choice for if_masked (RR, HR). Check if if_masked is in last dtd is in driver/xmlfiles and if transfer is done OK (SV).
For 2D grid (i.e for grids having a vertical dimension of one), nneighbour3D and trilinear should automatically result in nneighbour2D and bilinear respectively (PS: HR -ok the second 4 neighbours are the same as the first 4-; T: DD; for trilinear, we suppose that PS will give 2 times the same 4 neighbours)
Define an additional element in the smioc so that the user can choose the PSMILe behaviour in case a target point falls outside any source method cell (no interpolation, nearest-neighbour value, etc.) (SV); Use the value of this element in Ps (HR)
SMIOC: add if_masked element; modify Driver accordingly (SV)

novalue: if some of the 8 trilinear neighbours are masked, no neighbours are transfered from PS to T, and PRISM_undef value is given to that target point
tneighbour: if some of the 8 trilinear neighbours are masked, the non-masked points among those 8 points are used and transfered from PS to T; if the 8 trilinear neighbours are masked, PRISM_undef value is given to that target point
nneighbour: if some of the 8 trilinear neighbours are masked, the non-masked points among those 8 points are used; if the 8 trilinear neighbours are masked, the non masked nearest neighbour is used

2D1D:

Support of Reduced Gaussian grid (nearest-neighbour, bilinear, bicubic) (RR, HR for Ps; SV, JG for T)
2D1D can be used for all grids which vertical dimension can be expressed as z(k), i.e. for i.e. for source grid types PRISM_reglonlatvrt, PRISM_irrlonlat_regvrt, PRISM_unstructlatlon_regvrt; the mask may vary with depth. The interest of 2D1D is to allow a linear interpolation in the vertical. Change definition in ADSCCPMIODSMIOC.tex (SV).
Can be used for grids with vertical units into which linear interpolation makes sense.
2D1D combinations that work in Ps are: bilinear and linear, bilinear and none, nearest-neighbour and none. We could easily implement other 2D1D combinations but we then have to define new interface for Ps-T transfer (e.g. if 2D1D with bilinear and 8-nearest-neighbour in the vertical is requested).
Review 2D1D combinations in T; use SCRIP algorithm for oasis3-oasis4 reproductability (SV).
No need of particular PS-T transfer routines (except if we define other fancier vertical schemes)
T: Modify the weights calculation to really apply a 2D1D interpolation following user's choice in SMIOC (SV)

General:

Decide and maybe change mask convention, see mail from Rene dated June 24 (SV, DD, RR)
T: attribution of PRISM_undef value to masked target points or to target points for which no value can be calculated (DD)

License issues: OASIS4 should be delivered with a LGPL license. However, mpp_io which is included in OASIS4 uses a GPL license. Sophie will contact Balaji to see if the mpp_io version modified by Reiner (support of GMD-like grids) could be released with LGPL. If not, two versions of OASIS4 will have to be released: one LGPL without I/O, and one GPL with I/O. (SV Balaji agreed to release the mpp_io version for oasis3 and oasis4 with LGPL)

Transformer:

Parallelisation (TS, DD)
too much allocation/deallocation because of use of local structures? (DD, SV)

Definition of communicators if applications may contain many components running sequentially:

Check if ranks of components per application are needed (RR) (Since then, René confirmed that they are needed, 21/06)
If so, transfer ranks of components per application from Driver (DD)
Use ranks to define local component communicators (RR)
PS: Correct transfer of SMIOC information if application contains more than one component: currently, component information is overwritten. (RR, DD) (See proposition from René in mail dated June 22)
PSMILe support for two components running concurrently in one application ; test example provided in simple-cc (RR)
PSMILe support for two components running sequentially in one application; test example already provided in simple-cc (HR,RR)

Grid corner localisation:

update WEB site (RR)
check if shape of corner_xxx_array are coherent with what is expected (RR)
ordering: Fortran ordering if applicable; counterclockwise looking in direction of z at each level for irrlonlat_regvrt, irrlonlat_sigmavrt, reglonlat_sigmavrt, unstructlonlat_regvrt, unstructlonlat_sigmavrt; volume_type needed (see below in low priorities) for irrlonlatvrt and unstuctlonlatvrt

grid_type							nbr_c
reglonlatvrt	x(i)	y(j)	z(k)	c_1_a(i,lx)	c_2_a(j,ly)	c_3_a(k,lz)	lxlylz	lx=ly=lz=2
irrlonlat_regvrt	x(i,j)	y(i,j)	z(k)	c_1_a(i,j,lh)	c_2_a(i,j,lh)	c_3_a(k,lz)	lh*lz	lz=2
irrlonlatvrt	x(i,j,k)	y(i,j,k)	z(i,j,k)	c_1_a(i,j,k,lc)	c_2_a(i,j,k,lc)	c_3_a(i,j,k,lc)	lc
irrlonlat_sigmavrt	x(i,j)	y(i,j)	z(i,j,k)	c_1_a(i,j,lh)	c_2_a(i,j,lh)	c_3_a(i,j,k,lc)	2*lh
reglonlat_sigmavrt	x(i)	y(i)	z(i,j,k)	c_1_a(i,lx)	c_2_a(j,ly)	c_3_a(i,j,k,lc)	lc=2lxly
unstructlonlat_regvrt	x(nhor)	y(nhor)	z(k)	c_1_a(nhor,lh)	c_2_a(nhor,lh)	c_3_a(k,lz)	lh*lz	lz=2
unstructlonlat_sigmavrt	x(nhor)	y(nhor)	z(nhor,k)	c_1_a(nhor,lh)	c_2_a(nhor,lh)	c_3_a(nhor,k,lc)	lc=2*lh
unstuctlonlatvrt	x(ntot)	y(ntot)	z(ntot)	c_1_a(ntot,lc)	c_2_a(ntot,lc)	c_3_a(ntot,lc)	lc

Updates on PSMILe API WEB site (RR)

remove flow charts from WEB
remove prism_def_restvar
prism_def_var: keep var_nodims(2)for bundle only, dimension var_nodims(1) otherwise

Check if prism_get_localcomm called with PRISM_appl_id works fine (RR)

Redefine prism_def_partition: extent of blocks need to be added as arguments (SV, RV); update WEB(RR)

I/O:

Check fill_value: input grid point values that equal fill_value are turned to PRISM_undef in the PSMILe, and output grid point values that equal PRISM_undef are turned to fill_value in the output file (RV) (In T, data are received with appropriate type (MPI_Double, MPI_Real or MPI_Integer), then the interpolation is performed on the douple precision array ( CALL PRISMTrs_Interp(..., DBLE(rla_trans_out), ...) then the result is converted in the requested type, e.g. ila_trans_in = INT(dla_trans_in) )
memory requested with IO? SV checks what is the memory limit on elnino (-hinv) and reports to RV. RV traces memory requested by the different parts of OASIS4.

Driver:

Check that coupling fields are of same type on source and target sides (SV, DD; transients received by T are always stored into DOUBLE; they are converted to the target transient type as defined in the SMIOC before being sent to the target )
Check error bound checking (see René's mail June 16th)

XML files:

Sasa: DOUBLE read as INTEGER: use sscanf (DD)
Move sasa routines out of psmile_smioc.F90, put CPP flag around sasa calls in prism_init_comp, in order to avoid linking the application with libxml in non standalone case (SV)
Add reading of interp2D nneighbour2D bilinear and interp1D nneighbour1D linear SMIOC info (SV, DD)
SMIOC: change transient units to character strings; modify Driver accordingly; In general adapt Coupler and proto_ex to new smioc.dtd definition (SV)
grid_name (and not grid_family_name) should be coherent with PRISM_def_grid argument; change documentation and comments in code (SV)
Add bunvec for vector of bundles in transient_type which should be mandatory; modify Driver accordingly (SV)

Medium priority developments

Check PGF90 5.2 compilation (SV).
ESMF is working on improvement to support proleptic Gregorian before -4800. Test it when improvements are done. (RR).
Clean up prism.inc (ECMWF local codes)
Adapt Oasis4 and examples to PRISM SCE and SRE (after EU official release) (SV)
Interpolation:

Documentation on algorithms implemented in PSMILe (HR) (OK: I took a mental picture of the board in Utrecht)
3D nearest-neighbour (nneighbour3D) and trilinear (trilinear): PS: exact parallel neighbour search and use of user's choice for para_search value (para_search=precise or approximate ) (HR) (started but not for EU version).
2D conservativ2D (PS-T transfer of additional grid information (e.g. areas) in array(epio_size, nbr_info) will be needed - all these info will therefore be transfered with same precision (in T, use SCRIP algorithm for oasis3-oasis4 reproductability) (HR, DD)
2D bicubic (in T, use SCRIP algorithm for oasis3-oasis4 reproductability): the method with 16 points is implemented, validated for regular lat-lon grids; the method with gradient is implemented, and validated for regular lat-lon grid. The gradient method should work for any logically-rectangular grid, but has not been validated yet for other grids that regular lat-lon.

Workarounds to be solved with MP:

Translate comments in German (RR)
One workaround for gridless data

XML files

Write wrapping routines for transfer or direct reading in stand-alone case of SCC info, e.g. psmile_scc_init (transfer only application info) (DD; it is not possible to develop general wrapping routines; reading of SCC info in xml files in case of stand-alone needs to be done DD ).
AD: add element nbr_procs and arguments (lauching arguments) (SV)
SCC: transfer and use redirect attribute (DD, RR)
SMIOC:

use of pole_covered (RR)
remove useless elements in interp3D, interp2D, interp1D (SV)
Check if packing, scaling, adding is for NetCDF only (RV); if so, change SMIOC structure and doc (SV)

Interact with MetOffice for metadata definition (SV)

Interact with MetOffice: single model architecture (SV)
PSMILe function to get SCC and SMIOC info in model code (prism_get_persist, prism_query_var) (SV)
PSMILe calendar tool (SV)

Support of bundles, subgrids, vectors, and bundle of vectors for PS internal and DEL, PS I/O (bundle name need to be initialised OK) and T (definition of new PS-T transfer routines?, transfer of info on pole covered for vectors)

Support of non-gridded data

Support of other type of exchange_date (once, precise, lastdayinmonth, list)

Support of calendars other than proleptic Gregorian (RR). (Currently, in Ps proleptic Gregorian, 360, 365 (hardcoded)). Universal parameter such as calendar should be in described in AD, then specified in SCC. Then a routine prism_get_calendar should be written, usable by PS and applications (SV).

I/O:

Interaction with MetOffice; asynchronous IO server, netCDF vs HDF performance, etc.
Add option in SMIOC and use it in PS so that grid information is not written to the file with the data (RV, SV)
io_mode: parallel (use of parNetCFD)

Transformer:

Support of coupling fields with different types on source and target sides

Driver

Spawning of different applications on different hosts implemented; spawning of one application on different hosts should be OK but still to test
Check that interpolation method chosen is appropriate for the grid (e.g. check that vertical units are in meters if nneighbour3D or trilinear is chosen (SV)
Check that the different component SMIOC info is coherent (e.g. global parameters, transient units, sg_exch_date for a transient_out match the sg_exch_date for the corresponding transient_in, coupling frequency cannot be higher that the frequency of the prism_get, uniquess of application names...)

Coherence checks between information given through PSMILe API and SMIOC info: consistency checks only (e.g. grid info) or stop the run if incoherence is observed (i.e. coupling data)?

Check error code handling for all calls

Check prism_set_scalefactors (RR)
Implement prism_put_restart (RR)

Medium-Low priority developments

Support of stand-alone models (no Driver)

Implement prism_put_inquire, prism_iput, prism_iget, prism_wait

Additional Driver functions (prism_timestamp, prism_restartsaved, prism_message)

PS and T support of GME (one grid per block, only one variable name for many blocks)

Low priority developments

Interpolation

Interpolator-only mode: OASIS4 will not be usable in interpolator-only mode. Use of pseudo-models will be mandatory. Pseudo-models could be provided with OASIS4. Pseudo-models developed by Christian Mohn, presently National University of Ireland Galway, will be investigated (RR).
Storage of neighbours weights and address in file to avoid recalculation each time. Given OASIS4 design, this would not be trivial to implement; we leave this question open until the need is expressed.
3D conservativ3D, user3d
2D user2d
1D cubic, conservativ1D, user1d
1D vertical interpolation in log of vertical units (unit_logarithm )
reconstruction of vertical position of grid points based on the units and auxiliary fields (see CF convention)
1D1D1D:

Can be used for grids with vertical units into which linear interpolation makes sense.
1D1D1D supported in PSMILe following user's choice in SMIOC ( inter1D,interp1D,interp1D)
1D: 1Dnneighbour, linear, none
No need of particular PS-T transfer routine

Support of source_time_operation: tmin and tmax

Support of reduction, coupling and I/O after reduction

Support of algebraic_combinations (to be defined further)

Support of unstructured grid (PS I/O already OK if local partition can be expressed with only an offset in the global index space. For unstructured grids, this is the case if the local partition covers one segment in a 1D global space), expression of connectivity.

Volume type: for irrlonlatvrt and unstuctlonlatvrt grids, a description of the way the corners are connected is needed. An additional PSMILe call defining the connectivity between the corners could be used, but the prefered solution for now would be to explicitely defined different volume types in a User's Guide and refer to those explicitely in the SMIOC.

Saving of coupling restarts more frequently than run chuncks

Treatment of "abort restart"

Support of grid evolving with time

Automatic verification of what has changed and recalculation only for what has changed?
Problem if the grid changes between prism_put and coreesponding prism_get with lag
Problem of automatic writing and reading of coupling restarts

XML files

SMIOC:

compute_space not needed for now neither in SMIOC, Driver, PSMIle (corresponding ig_compute_id transfered but not defined).
conditional_computation not needed for now neither in SMIOC, Driver, PSMILe

SCC keyword for saving more frequently than run chuncks

PSMILe C API

prism_def_var: arguments redundant with SMIOC information (SV)

PSMILe I/O:

Format supported: mpp_ieee32 OK for I; mpp_native for I; mpp_ascii; GRIB; other
Output file size chosen by user; add a keyword in SMIOC (RV, SV)