.

ACKNOWLEDGMENTS

The development of this new version of OASIS, OASIS3-MCT v6 has been possible thanks the following fundings:

We would like to thank the main past or present developers of OASIS (in alphabetical order, with the name of their institution at the time of their contribution to OASIS):

Arnaud Caubel (LSCE/IPSL & FECIT/Fujitsu), Laure Coquart (CNRS/CERFACS) Anthony Craig (CERFACS - consultant), Damien Declat (CERFACS), Italo Epicoco (CMCC), Rupert Ford (STFC), Philippe Gambron (STFC), Veronika Gayler (MPI-M&D), Josefine Ghattas (CERFACS), Christopher Goodyer (NAG), Moritz Hanke (DKRZ), Gabriel Jonville (CERFACS), Jean Latour (CERFACS & Fujitsu-Fecit), Eric Maisonnave (CERFACS), Silvia Mocavero (CMCC), Andrea Piacentini (CERFACS - consultant) Elodie Rapaport (CERFACS), Sami Saarinen (ECMWF), Eric Sevault (Météo-France), Laurent Terray (CERFACS), Olivier Thual (CERFACS), Sophie Valcke (CERFACS), Reiner Vogelsang (SGI Germany), Li Yan (CERFACS).

We also would like to thank the following people for their help and suggestions in the design of the OASIS software (in alphabetical order, with the name of their institution at the time of their contribution to OASIS):

Dominique Astruc (IMFT), Chandan Basu (NSC, Sweden), Sophie Belamari (Météo-France), Dominique Bielli (Météo-France), Yamina Boumediene (CERFACS), Gilles Bourhis (IDRIS), Pascale Braconnot (IPSL/LSCE), Sandro Calmanti (Météo-France), Christophe Cassou (CERFACS), Yves Chartier (RPN), Jalel Chergui (IDRIS), Philippe Courtier (Météo-France), Philippe Dandin (Météo-France), Michel Déqué (Météo-France), Ralph Doescher (SMHI), Jean-Louis Dufresne (LMD), Jean-Marie Epitalon (CERFACS), Laurent Fairhead (LMD), Uwe Fladrich (SMHI), Marie-Alice Foujols (IPSL), Gilles Garric (CERFACS), Christopher Goodyer (NAG), Eric Guilyardi (CERFACS), Charles Henriet (CRAY France), Pierre Herchuelz (ACCRI), Maurice Imbard (Météo-France), Luis Kornblueh (MPI-M), Stephanie Legutke (MPI-M&D), Claire Lévy (LODYC), Yann Meurdesoif (IPSL/LSCE) Olivier Marti (IPSL/LSCE), Sébastien Masson (IPSL/LOCEAN) Claude Mercier (IDRIS), Pascale Noyret (EDF), Marc Pontaud (Météo-France), Adam Ralph (ICHEC), René Redler (MPI-M), Hubert Ritzdorf (CCRLE-NEC), Tim Stockdale (ECMWF), Jan Streffing (AWI), Rowan Sutton (UGAMP), Véronique Taverne (CERFACS), Jean-Christophe Thil (UKMO), Nils Wedi (ECMWF).

1 Introduction

In 1991, CERFACS started the development of a software interface to couple existing ocean and atmosphere numerical General Circulation Models. Today, different versions of the OASIS3-MCT coupler are used by at least 65 modelling groups all around the world to couple more than 80 applications on different computing platforms1. OASIS3-MCT sustained development is ensured by a collaboration between CERFACS and the Centre National de la Recherche Scientifique (CNRS) and its maintenance and user support is regularly reinforced with additional resources coming from European and national projects.

OASIS3-MCT is a portable set of Fortran 77, Fortran 90 and C routines. Low-intrusiveness, portability and flexibility are OASIS3-MCT key design concepts. After compilation OASIS3-MCT is a coupling library to be linked to the component models, and which main function is to interpolate and exchange the coupling fields between the components to form a coupled system. OASIS3-MCT supports coupling of 2D logically-rectangular fields but 3D fields and 1D fields expressed on unstructured grids are also supported using a one dimension degeneration of the structures.

Internally, OASIS3-MCT uses MCT, the Model Coupling Toolkit2 , developed by the Argonne National Laboratory in the USA. MCT implements fully parallel remapping, as a parallel matrix vector multiplication on the set of source or target component processes, and parallel distributed exchanges of the coupling fields via Message Passing Interface (MPI). Its design philosophy, based on flexibility and minimal invasiveness, is close to the OASIS3-MCT approach.

To communicate with another component, a component model needs to include few specific calls to OASIS3-MCT Fortran Application Programming Interface (API). OASIS3-MCT also has python and C language bindings. For remapping the coupling fields from the source to the target grid, OASIS3-MCT v6 includes a modified version of the SCRIP library and is interfaced with YAC . OASIS3-MCT also supports basic non-parallel file I/O using NetCDF . The configuration of the coupling exchanges needs to be described externally by the user in a text file, the namcouple.

The developments realised between the different versions of OASIS3-MCT are described in Appendix 8.

We believe that OASIS3-MCT will still provide an efficient and easy-to-use coupling solution for many climate modelling groups in the few years to come.

1.1 Step-by-step use of OASIS3-MCT

To use OASIS3-MCT for coupling codes, one has to follow these steps:

  1. Obtain OASIS3-MCT source code (see chapter 1.2).

  2. Get familiar with OASIS3-MCT, either by following the Short Private Online Course (SPOC, see https://cerfacs.fr/en/online-traning-sessions/ ) or by going through the tutorial steps. A tutorial is also available in directory examples/tutorial_communication and all explanations are provided in the document tutorial_communication.pdf therein.

  3. Identify the coupling or I/O fields and adapt the codes to implement the coupling exchanges with the OASIS3-MCT coupling library based on MPI message passing. The OASIS3-MCT coupling library uses NetCDF and therefore can also be used to perform I/O actions from/to disk files. For more detail on how to use the OASIS3-MCT API in the codes, see chapter 2.

  4. Define all coupling and I/O parameters and transformations required to adapt each coupling field from the source model grid to the target model grid. On this basis, prepare OASIS3-MCT configuring file namcouple. OASIS3-MCT supports different interpolation algorithms as described in chapter 4. Remapping files can be computed online using the SCRIP or YAC library, or offline using either the SCRIP, YAC, ESMF or XIOS (see examples/regrid_environment, section 6.3.3) and read in during the run using the MAPPING transformation.

    We strongly recommend to tests the quality of the chosen transformations and remappings off-line using the environment available in examples/regrid_environment and explanations provided in the document regrid_environment.pdf therein.

  5. Generate required auxiliary data files , if needed (see chapter 5).

  6. Compile OASIS3-MCT, the component models and start the coupled experiment. For details on how to compile and run a coupled model with OASIS3-MCT, see section 6.

If you need extra help, do not hesitate to contact us at oasishelp(at)cerfacs.fr .

1.2 OASIS3-MCT sources

OASIS3-MCT sources are available from CERFACS git server. To obtain more detail on downloading the sources, please fill in the registration form at https://oasis.cerfacs.fr/en/downloads/ .

OASIS3-MCT directory structure is the following one:

 - oasis3-mct/lib/cbindings        C language bindings source code
                 /mct              Model Coupling Toolkit Coupling Software
                 /psmile           OASIS3-MCT coupling library
                 /scrip            SCRIP remapping library  
                 /yac              YAC  remapping library 
                 /yaxt	            communication layer used by YAC            

 - oasis3-mct/doc                  OASIS3-MCT User Guide

 - oasis3-mct/util/make_dir        Utilities to compile OASIS3-MCT
                  /load_balancing  Tool for load balancing analysis
                  /env_tests        Script to compile and launch toy models          
                  
 - oasis3-mct/pyoasis	             Python wrapper source code

 - oasis3-mct/examples             Environment to compile, run and use
                                   different toy coupled models.

1.3 Licenses and Copyrights

Copyright © 2025 Centre Européen de Recherche et Formation Avancée en Calcul Scientifique (CERFACS).

This software and ancillary information called OASIS3-MCT is free software. CERFACS has rights to use, reproduce, and distribute OASIS3-MCT. The public may copy, distribute, use, prepare derivative works and publicly display OASIS3-MCT under the terms of the Lesser GNU General Public License (LGPL) as published by the Free Software Foundation, provided that this notice and any statement of authorship are reproduced on all copies. If OASIS3-MCT is modified to produce derivative works, such modified software should be clearly marked, so as not to confuse it with the OASIS3-MCT version available from CERFACS.

The developers of the OASIS3-MCT software are researchers attempting to build a modular and user-friendly coupler accessible to the climate modelling community. Although we use the tool ourselves and have made every effort to ensure its accuracy, we can not make any guarantees. We provide the software to you for free. In return, you –the user– assume full responsibility for use of the software. The OASIS3-MCT software comes without any warranties (implied or expressed) and is not guaranteed to work for you or on your computer. Specifically, CERFACS and the various individuals involved in development and maintenance of the OASIS3-MCT software are not responsible for any damage that may result from correct or incorrect use of this software.

1.3.2 MCT copyright statement

Modeling Coupling Toolkit (MCT) Software

Copyright © 2021, UChicago Argonne, LLC as Operator of Argonne National Laboratory. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  3. The end-user documentation included with the redistribution, if any, must include the following acknowledgment: "This product includes software developed by the UChicago Argonne, LLC, as Operator of Argonne National Laboratory." Alternately, this acknowledgment may appear in the software itself, if and wherever such third-party acknowledgments normally appear.

    This software was authored by:

  4. WARRANTY DISCLAIMER. THE SOFTWARE IS SUPPLIED "AS IS" WITHOUT WARRANTY OF ANY KIND. THE COPYRIGHT HOLDER, THE UNITED STATES, THE UNITED STATES DEPARTMENT OF ENERGY, AND THEIR EMPLOYEES: (1) DISCLAIM ANY WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-IN- FRINGEMENT, (2) DO NOT ASSUME ANY LEGAL LIABILITY OR RESPONSIBILITY FOR THE ACCURACY, COMPLETENESS, OR USEFULNESS OF THE SOFTWARE, (3) DO NOT REPRESENT THAT USE OF THE SOFTWARE WOULD NOT INFRINGE PRIVATELY OWNED RIGHTS, (4) DO NOT WARRANT THAT THE SOFTWARE WILL FUNCTION UNINTERRUPTED, THAT IT IS ERROR-FREE OR THAT ANY ERRORS WILL BE CORRECTED.

  5. LIMITATION OF LIABILITY. IN NO EVENT WILL THE COPYRIGHT HOLDER, THE UNITED STATES, THE UNITED STATES DEPARTMENT OF ENERGY, OR THEIR EMPLOYEES: BE LIABLE FOR ANY INDIRECT, INCIDENTAL, CONSEQUENTIAL, SPECIAL OR PUNITIVE DAMAGES OF ANY KIND OR NATURE, INCLUDING BUT NOT LIMITED TO LOSS OF PROFITS OR LOSS OF DATA, FOR ANY REASON WHATSOEVER, WHETHER SUCH LIABILITY IS ASSERTED ON THE BASIS OF CONTRACT, TORT (INCLUDING NEGLIGENCE OR STRICT LIABILITY), OR OTHERWISE, EVEN IF ANY OF SAID PARTIES HAS BEEN WARNED OF THE POSSIBILITY OF SUCH LOSS OR DAMAGES.

1.3.3 SCRIP 1.4 license copyright statement

Copyright © 1997, 1998 the Regents of the University of California. This software and ancillary information (herein called SOFTWARE) called SCRIP is made available under the terms described here. The SOFTWARE has been approved for release with associated LA-CC Number 98-45. Unless otherwise indicated, this SOFTWARE has been authored by an employee or employees of the University of California, operator of Los Alamos National Laboratory under Contract No. W-7405-ENG-36 with the United States Department of Energy. The United States Government has rights to use, reproduce, and distribute this SOFTWARE. The public may copy, distribute, prepare derivative works and publicly display this SOFTWARE without charge, provided that this Notice and any statement of authorship are reproduced on all copies. Neither the Government nor the University makes any warranty, express or implied, or assumes any liability or responsibility for the use of this SOFTWARE. If SOFTWARE is modified to produce derivative works, such modified SOFTWARE should be clearly marked, so as not to confuse it with the version available from Los Alamos National Laboratory.

1.3.4 YAC license copyright statement

Copyright (c) 2025 The YAC Authors

SPDX-License-Identifier: CC-BY-4.0

2 Interfacing a component code with OASIS3-MCT

At run-time, OASIS3-MCT performs parallel exchange of coupling data between parallel components and sub-components and allows regridding (also called remapping or interpolation), time integration or accumulation and other transformations of these coupling fields.

This chapter describes how to adapt the component codes to couple them through OASIS3-MCT.

OASIS3-MCT supports coupling exchanges between parallel components and sub-components deployed in diverse configurations. The different possibilities and how to use the OASIS3-MCT library accordingly are described in section 2.1.

The OASIS3-MCT Application Programming Interface (API) includes different classes of modules or routines that are described in detail in sections 2.2, 2.3 and 2.4 for Fortran, C and python codes respectively. In section 2.5, the reader will also find an overview of the MCT library (see 2.5.4) and additional notes on how to exchange scalars (see 2.5.1), how to to reproduce different coupling algorithms with OASIS3-MCT using the LAG index (see 2.5.2), and on the SEQ index (see 2.5.3).

2.1 Configurations of components supported

Since OASIS3-MCT_3.0 release, coupling exchanges between components and sub-components can be deployed in a much larger diversity of configurations than before. This is illustrated on figure 2.1 and how to use the OASIS3-MCT library accordingly is detailed on figure 2.2. All OASIS3-MCT API routines are also described in details in sections 2.2, 2.3 and 2.4.

We call here an “executable” a compiled code forming a part of or the whole coupled system. A “component” is the ensemble of processes, or tasks, within the coupled system calling oasis_init_comp with the same comp_name argument (see section 2.2.2.1). A “sub-component” is the subset of tasks within a component sending or receiving coupling fields on a specific grid. Of course, a component may have only one sub-component that gathers all its tasks.

In examples/tutoria_communication, examples/regrid_environment and
pyoasis/examples, one finds practical examples of how to use the OASIS3-MCT library (see sections 6.3.1 and 6.3.3).

With OASIS3-MCT, it is currently possible to implement coupling exchanges (in the following paragraph, the text between [ and ] refers to figure 2.1) :

The different configuration of components supported by OASIS3-MCT. Two executables exe1 and exe2 are running concurrently on separate sets of MPI tasks (0-5 for exe1 and 6-37 for exe2). Executable exe1 includes only one component comp1 that has coupling fields defined on only one grid grid1 (decomposed on all of its 6 tasks). Executable exe2 includes 3 components, comp2, comp3, and comp4 running concurrently respectively on tasks 6-11, 12-33 and 34-37. Component comp2 participates in the coupling with fields defined on only one coupling grid grid2 (decomposed on all of its 5 tasks) while comp4 does not participate at all in the coupling. Component comp3 has 3 sub-components, respectively exchanging coupling fields defined on grid3 (tasks 12-21), grid4 (tasks 22-30) and grid5 (tasks 12-26, therefore overlaping with both grid3 and grid4); finally, comp3 has 3 tasks (31-33) not involved in the coupling. Sub-componentsexe2-comp3-grid3 and exe2-comp3-grid5, or sub-components exe2-comp3-grid4 and exe2-comp3-grid5 are examples of coupling between sub-components running sequentially on overlapping sets of tasks.
The sequence of OASIS3-MCT calls that have to be implemented in the codes so to allow the configuration of components described on figure 2.1. Each MPI tasks has to call oasis_init_comp once with the name of its component as \(2^{nd}\) argument and oasis_terminate before exiting. As none of comp4 tasks is participating to the coupling, comp4 tasks calls oasis_init_comp with coupled=.false.” as \(4^{th}\) argument and does not call any other OASIS3-MCT routine except oasis_terminate. As some of comp3 tasks are participating in the coupling, all comp3 tasks have to call oasis_init_comp, oasis_get_localcomm, oasis_create_couplcomm, oasis_enddef and oasis_terminate (these are the only routine to be called by comp3 tasks 31-33 not participating to the coupling). To initialise the coupling exchanges, the tasks of a sub-component holding a field decomposed on a specific grid have to call the oasis_def_partition to express the decomposition of the grid, oasis_def_var to declare the coupling field and oasis_enddef. Finally, the tasks of a sub-component exchanging coupling fields have to call oasis_put and oasis_get accordingly.

The sequence of OASS3-MCT API routines that have to be called in the different cases is detailed on figure 2.2. These routines are also described in detail in the next section.

2.2 OASIS3-MCT Fortran API

To interact with the rest of the coupled system, few calls of the OASIS3-MCT library routines, which sources can be found in oasis3-mct/lib/psmile directory, have to be implemented in the Fortran codes of the components. The interface for C or C++ codes and for python codes are described in section 2.3 and section 2.4 respectively. They belong to the following classes:

  1. Module to use (section 2.2.1)

  2. Initialisation (section 2.2.2)

  3. Partition definition (section 2.2.3)

  4. Grid data file definition (section 2.2.4)

  5. Coupling-I/O field declaration (section 2.2.5)

  6. End of definition phase (section 2.2.6)

  7. Coupling-I/O field sending and receiving (section 2.2.7)

  8. Termination (section 2.2.8)

  9. Optional auxiliary routines (section 2.2.9)

2.2.1 Module to use in the code

To use OASIS3-MCT library, a user needs to add in his code:

Use of module mod_oasis provides access to all methods required that will continue to evolve in the future, always ensuring backward compatibility. In the following sections, the interface names are defined and the description of the interface arguments is provided.

2.2.2 Initialisation

2.2.2.1 Coupling initialisation

2.2.2.2 Communicator for internal parallelisation

If this communicator already exists in the code, the component should simply provide it to OASIS3-MCT with:

2.2.3 Partition definition

The coupling fields sent or received by a component are usually scattered among the different component processes. With OASIS3-MCT, all processes exchanging coupling data have to describe, in a global index space spanning the full domain, the local partitioning of the different grids onto which the data is expressed (see 2.2.4 for the grid definition).

The choices detailed in the following provide a compact description of 2D partitions. In addition, a special case was introduced in OASIS3-MCT v6 to facilitate the description of 3D partitions. This does not mean that calculation of 3D interpolation weights are supported by OASIS3-MCT. In particular, the cube partition (see below) requires pre-computed 3D interpolation weights and forbids any other pre- or post-processing transformation that supposes a 2D structure of the exchanged data.

The processes not implied in the coupling do not have to call this routine (for backward compatibility with OASIS3-MCT_2.0, they may still call it describing a null partition, i.e. with ig_paral(:)=0).

The partitions for the different processes cannot overlap, i.e. haloes are not supported and it is not possible to define a partition across multiple active OpenMP threads.3.

2.2.3.1 Serial (no partition)

This is the choice for a grid entirely supported by only one process . In this case, we have ig_paral(1:3):

2.2.3.2 Apple partition

Each partition is a segment of the global domain, described by its global offset and its local size. In this case, we have ig_paral(1:3):

Figure 2.3 illustrates an Apple partition over 3 processes.

Apple partition. We assume here that the global index starts at 0 in the upper left corner.

2.2.3.3 Box partition

Each partition is a rectangular region of the global domain, described by the global offset of its upper left corner, and its local extents in the X and Y dimensions. The global extent in the X dimension must also be given. In this case, we have ig_paral(1:5):

Figure 2.4 illustrates a Box partition over 3 processes.

Box partition. We assume here that the global index starts at 0 in the upper left corner.

2.2.3.4 Orange partition

Each partition is an ensemble of segments in the global domain. Each segment is described by its global offset and its local extent. In this case, we have ig_paral(1:N) where N = 2 + 2*number of segments

Figure 2.5 illustrates an Orange partition with 3 segments for one process. The other process partitions are not illustrated.

Note that each partition has to be defined with monotonically increasing global index values.

Orange partition for one process. We assume here that the global index starts at 0 in the upper left corner.

2.2.3.5 Points partition

This partition is a list of global indices associated with each process. The index naming is arbitrary but must be consistent between all processes involved in the partition description. In this case, we have ig_paral(1:N) where N = 2 + number of points. As for Orange, each partition has to be defined with monotonically increasing global index values.

2.2.3.6 Cube partition

For the special case of 3D fields interpolation with user-provided remapping weights, OASIS3-MCT supports a partition which acts as a Box partition at every vertical level without the need of repeating it. On every 2D level, each local partition is a rectangular region of the 2D global domain, described by the global offset of its upper left corner, and its local extents in the X and Y 2D dimensions. The global extents in the X and Y 2D dimensions and the number of vertical levels must also be given. In this case, we have ig_paral(1:7):

This type of partition is adapted to specific couplings, e.g. horizontal coarsening in the exchanges between a high-resolution ocean model and a lower-resolution bio-geo-chemistry model. An example of how to use this type of partition is given in /pyoasis/examples/14-test-inter3d.

2.2.4 Grid data file definition

Grid data files are required by OASIS3-MCT for specific operations, see sections 4 and 5.1. These grid data files can be created by the user before the run or can be written directly at run time by the components with the routines described in this section. In that last case, the corresponding routine will create a grid data files if it does not exist. If the grid data file does exist, the routine can still be used to add grid definition in that file, but it will not overwrite a grid definition if it already exist in the file with the same grid name.

These routines can be called only by one component process to write the whole grid or by each process holding a part of a grid. In the former case, optional argument il_part_id is not needed and the arrays handling the longitudes of the grid points or corners, the latitudes of the grid points or corners, the masks, fractions and areas of the grid cells need to cover the whole grid. In the later case, the il_part_id returned by oasis_def_partition needs to be provided as input argument and the arrays need to cover only the local partition of the grid.

The field names in the grids.nc, masks.nc, and areas.nc follow a well-defined convention (see also section 5.1). The fields are two-dimensional, and each field name consists of a grid acronym followed by a string that identifies the field. For instance, the center latitudes and longitudes for the grid torc will be called respectively torc.lat and torc.lon. The grids.nc file contains the center latitudes (.lat) and longitudes (.lon) as well as the corner latitudes (.cla) and corner longitudes (.clo). The corner fields have a third dimension associated with the number of corners per gridcell. The area.nc file contains the area field (.srf). The masks.nc file contains the mask (.msk) and static fraction (.frc) fields.

Note that if all remappings are based on user-defined weights, the files grids.nc and areas.nc are not required, while masks.nc is only necessary for the CHECKIN or CHECKOUT diagnostics (see sections 4).

The creation of the different grid data files is completed in the routine oasis_enddef.

2.2.5 Coupling field declaration

All processes of a component that send or receive a coupling field, or a part of it, needs to declare the coupling field.

Processes not implied in the coupling do not have to call this routine at all but for backward compatibility with OASIS3-MCT_2.0, they may still call it with any name and il_part_id.

2.2.6 End of definition phase

All processes of components at least partly involved in the coupling (e.g. comp3 in figure 2.2) have to close the definition phase. Different configurations of components and corresponding use of oasis_enddef are described in section 2.1 and on figures 2.1 and 2.2.

2.2.7 Sending “put” and receiving “get” actions

This section describes how to send (put) and receive (get) fields through OASIS-MCT API. This coupling interface supports several ranks and types of coupling fields. The field decomposition must be consistent with the grid partition (see 2.2.3)5 and the bundle fields are supported (i.e. have an extra non-spatial dimension, for example for different ice categories).

The bundle field option can be used to send or receive together multi-level variables, multiple related fields, and other types of fields. The fields must share a common partition and common namcouple settings (e.g. interpolation) to be bundled. While this is a useful feature for multi-level fields, this does not mean that 3D interpolation is supported. Each field in the bundle is treated internally as a separate field in the coupling layer without any information about the relationship between the fields in the bundle. In fact, the bundle field variables are internally renamed and a field number is appended to the variable name to keep track of the distinct fields in the bundle. That updated variable name will appear in restart and output files.

It is possible to define a 1D bundle or 2D bundle field with a bundle dimension of 1. So in general, fld, the field passed into the get and put interface can have rank 1, 2, or 3 and includes the following possible options:

For a given bundle, the number of fields must match in the source and target models. This is explicitly checked within the coupling layer and will lead to an abort if the match is not effective.

2.2.7.1 Sending a coupling field (or writing a coupling restart file)

In the component time step loop, each process sends its part of the coupling (or I/O) field.

To ensure a proper use of the oasis_put, one has to take care of the following aspects:

2.2.7.2 Receiving a coupling (or I/O) field

In the component time stepping loop, each process receives its part of the coupling field.

To ensure a proper use of the oasis_get, one has to take care of the following aspects:

2.2.8 Termination

2.2.9 Auxiliary routines

The following auxiliary routines are currently available.

2.3 OASIS3-MCT C API

OASIS3-MCT is distributed with C bindings and can be called from models written in C and in C++. These bindings leverage the Fortran ISO_C_BINDING standard. The C bindings can be compiled into static or shared libraries by the OASIS3-MCT TopMakefileOasis3 as documented in 6.1.

The C interfaces largely match up with equivalent interfaces in Fortran. An interface named oasis_interface in Fortran can be expected to be named oasis_c_interface in C.

All of the C interfaces return an integer error code which can be tested against the OASIS_Ok (or the equivalent OASIS_Success) constant. The OASIS_CHECK_ERR macro aborts OASIS3-MCT with a meaningful message in the debug files in case of failure. For most of the functions, the return code is consistent with the kinfo argument in the Fortran interfaces (see section 2.2). However, for the put and get communication functions, the return error code only indicates success or failure, while the detailed status is returned by the kinfo argument (equivalent to the returned value of info argument for the Fortran oasis_put and oasis_get routines, see section 2.2.7.1).

For example, the following constructs are equivalent in the two languages:

call oasis_get_localcomm(localcomm, kinfo)
if (kinfo .ne. OASIS_Ok)  &
   &  call OASIS_Abort(comp_id, "oasis_get_localcomm", &
   &  "Runtime error", __FILE__, __LINE__)

and

OASIS_CHECK_ERR(oasis_get_localcomm(localcomm))

For convenience a similar macro OASIS_CHECK_MPI_ERR has been defined for testing the return code of any MPI function against MPI_SUCCESS and cleanly aborting OASIS3-MCT in case of failure.

Use of these C bindings are illustrated in practical examples in directories /C in the different subdirectories in pyoasis/examples/. Notes and deviations from the Fortran standard are noted below.

2.4 OASIS3-MCT python API

The source code of pyOASIS is in the directory pyoasis/src. Complete documentation is available in pyoasis/pyoasis.pdf. The pyoasis interface ultimately call the Fortran version. This is done by wrapping the Fortran in ISO-C bindings (see lib/cbindings/fortran_isoc), then wrapping the Fortran ISO-C bindings in C (see lib/cbindings/c), then wrapping the C bindings in python (see pyoasis/src). This method provides both the C and python bindings. Examples on how to use pyOASIS are provided in pyoasis/examples. To run all tests including python, C, and Fortran examples, use make test. Make sure that the component models, OASIS3-MCT, and all python modules used are compiled with the same compiler. The python wrapper functions are briefly described next.

2.4.1 Fortran python API correspondence

Figures 2.6 , 2.7 and 2.8 show the Fortran python API correspondence for different parts of the API. Different examples implementing the different parts of the API with the Fortran, C and python interfaces are also provided as practical illustrations in directory pyoasis/examples and are described in section 6.3.4.

Fortran python AP correspondence for the initialisation, communication and partition definition.
Fortran python AP correspondence for the grid definition, variable declaration and end of definition phase.
Fortran python AP correspondence for the coupling field exchanges, termination and auxialiary routines.

2.5 Additional notes on coupling functionality

2.5.1 Coupling scalar values

OASIS3-MCT does not have a distinct feature to support scalar coupling. By scalar coupling, we mean scalar variables such as date and time, logical flags, integer or real parameters, or other scalar data that might be defined identically across all MPI tasks in a component or even just on the root task or a subset of tasks. It is often desirable to communicate scalar data between components to coordinate scientific or technical features. There isn’t a feature that supports this type of coupling specifically, but it is possible to do so using available interfaces. The most robust implementation is to setup root to root coupling of scalars between components. To do so :

This can be extended as needed to send or receive from non root processes. But one cannot send the scalar data to multiple processes as this violates the halo restriction in MCT.

2.5.2 The lag concept

Using the OASIS3-MCT coupling library, the user has the flexibility to reproduce different coupling algorithms defining LAG values for the different coupling fields . See section 3.3 for more details on how to specify a LAG for a coupling field. In the components, the sending and receiving routines, respectively oasis_put and oasis_get, can be called at each component timestep, with the appropriate date argument giving the actual time (at the beginning of the timestep), expressed in number of seconds since the start of the run, or in any other time units as long as the same are used in all components and in the namcouple (see section 2.2.7.1). This date argument is automatically analysed by the coupling library and, depending on the coupling period and the lag chosen by the user for the coupling field in the namcouple (LAG), different coupling algorithms can be reproduced without modifying the component codes themselves.

The lag (LAG) must be expressed in the time unit used and can be positive or negative but should never be larger (in absolute magnitude) than the coupling period of any field due to problems with restartability and dead-locking. When a component calls a oasis_put, the value of the lag is automatically added to the value of the date argument and the “put” is actually performed when the sum date+lag is a coupling time. In the target component, this “put” will match a oasis_get for which the date argument is that same coupling time. So the lag only shifts the time at which the data is sent but not the time at which the data is received.

When the lag is positive, a restart file must be available to initiate the coupling. See section 5.2 for details on the structure of the coupling restart files. For a field with positive lag, the source component automatically reads the field in the restart file during the coupling initialization phase (below the oasis_enddef) and send the data to match the oasis_get performed at time=0 in the target component. The final coupling data on the source side will then be automatically written to the restart file for use in the next run6.

On figure 2.9, short black arrows correspond to oasis_put or oasis_get called in the component that do not lead to any “put" or receiving action; long black arrows correspond to oasis_put or oasis_get called in the components that do actually lead to a “put" or “get” action; long red arrows correspond to oasis_put or oasis_get called in the component models that lead to a reading or writing of the coupling field from or to a coupling restart file.

A coupling algorithm exploiting the LAG concept, is illustrated on figure 2.9.

LAG concept first example

During a coupling timestep, model A receives \(F_2\) and then sends \(F_1\); its timestep length is 4. During a coupling timestep, model B receives \(F_1\) and then sends \(F_2\); its timestep length is 6. \(F_1\) and \(F_2\) coupling periods are respectively 12 and 24. If \(F_1\)/\(F_2\) “put" action by model A/B was used at a coupling timestep to match the model B/A “get" action, a deadlock would occur as both models would be initially waiting on a “get" action. To prevent this, \(F_1\) and \(F_2\) produced at the timestep before have to be used to match respectively the model B and model A “get" actions.

This implies that a lag of respectively 4 and 6 seconds must be defined for \(F_1\) and \(F_2\). For \(F_1\), the oasis_put performed at time 8 and 20 by model A will then lead to “put" actions (as 8 + 4 = 12 and 20 + 4 = 24 which are coupling periods) that match the “get" actions performed by oasis_get called by model B at times 12 and 24. For \(F_2\), the oasis_put performed at time 18 by model B then leads to a “put" action (as 18 + 6 = 24 which is a coupling period) that matches the “get" action performed at time 24 by the oasis_get called by model A.

At the beginning of the run, as their LAG index is greater than 0, the first oasis_get of \(F_1\) and \(F_2\) will automatically be fulfilled with fields read from their respective coupling restart files. The user therefore has to create those coupling restart files before the first run in the experiment. At the end of the run, \(F_1\) having a lag greater than 0, is automatically written to its coupling restart file below the last \(F_1\) oasis_put as the date+lag equals the total run time. The analogue is true for \(F_2\). These coupling restart fields will automatically be read in at the beginning of the next run below the respective oasis_get.

In conclusion, using appropriate LAG indices ensures that the target component receives the coupling fields produced by the source component at a previous time. This is, in some coupling configurations, essential to avoid deadlock situations.

2.5.3 The sequence concept

The order of coupling operations in the system is determined solely by the order of calls to send (oasis_put or “put”) and receive (oasis_get or “get”) data in the components in conjunction with the setting of the LAG in the namcouple. Receiving data is always blocking while sending data is non-blocking with respect to the component making that call. It is possible to deadlock the system if the relative orders of puts and gets in different components are not compatible.

The sequence (SEQ) index in the namcouple file now provides the coupling layer with an ability to detect a deadlock before it happens and exit. It does this by tracking the order of get and put calls in components compared to the SEQ specified in the namcouple. If there are any inconsistencies, the component will abort gracefully with a useable error message before the system deadlocks. If there are any coupling dependencies in the system, use of the SEQ index is recommended for diagnosis but has no impact on the ultimate solution and is NOT required.

In the following two examples, there are two models, each one sending a field to the other at every coupling period without any lags. In the first case, there is no dependency as each model first sends and then receives some data.

     model1        model2
     ------        ------
    put(fld1)     put(fld2)
    get(fld2)     get(fld1)

In this case, there is no sequencing dependency and specifying SEQ indices is not needed7.

In the next example, there is a dependency in the sequencing.

     model1        model2
     ------        ------
    put(fld1)     get(fld1)
                  fld2=g(fld1)
    get(fld2)     put(fld2)

In model2, fld2 depends on fld1. If SEQ is not used and if, for example, model1 does not have the consistent ordering of the put and get shown above (required by model2), then the models would deadlock and hang. If this dependency is known, there is a benefit in setting SEQ=1 for fld1 and SEQ=2 for fld2; at runtime. If the sequencing of model1 or model2 does not match the above diagram, then the coupling layer will detect it and will exit gracefully with an error message.

Again, the SEQ namecouple setting is only diagnostic and is not required.

2.5.4 A brief overview of MCT

OASIS3-MCT leverages the MCT coupling library developed at Argonne National Laboratory. That library is designed to couple fields on static grids between model components. The fields can be decomposed across mutiple processes, and the decomposition and number of proceses involved can be arbitrary in each component.

MCT also provides the ability to map (i.e. interpolate or regrid) data between grids as long as the interpolation is linear and can be computed by a linear sparse matrix multiply. MCT does not compute interpolation weights, but it has interfaces that allow those weights to be passed into MCT. Within MCT, mapping and communication are also treated as independent features.

OASIS3-MCT supports both mapping and communication of data through MCT. A coupling field can be interpolated to the destination grid on the source processes then communicated to the destination component; or a coupling field can be communicated to the destination component on the source grid then mapped to the destination grid on the destination component (see $MAPLOC in section 4.3). At the present time, it is not possible to map the field as part of the communication rearrangement, although that capability should be possible to implement. The separation of mapping and communication is handled by the OASIS layer. Users only need to be aware of a few options that can be set to fine tune the performance of these operations.

OASIS3-MCT also imposes a few other constraints on the usage. For example, MCT does not support haloed communication. The partitions defined by the component that describes the field decomposition cannot reference the same global grid cell more than once.

Also, neither OASIS3-MCT nor MCT support dynamically varying grids nor dynamically varying decompositions at the present time.

Finally, MCT itself, and therefore the communication layer in OASIS, has no OpenMP support. Users are discouraged from calling OASIS3-MCT oasis_put and oasis_get from a threaded region. And it is not possible to define a partition (decomposition) across multiple active threads. The work that OASIS3-MCT does related to decomposition rearrangement for communication or mapping per se is MPI-based. There are some floating point operations in the sparse matrix multiple as well as in some of the OASIS3-MCT transformation operations; but generally, these are highly parallel and would often not benefit from use of OpenMP. The only use of OpenMP in OASIS3-MCT is within the mixed MPI+OpenMP parallel version of the SCRIP library for the calculation of the remapping weights (see section 4.3).

3 The configuration file namcouple

The OASIS3-MCT configuration file namcouple contains, below pre-defined keywords, all user-defined information necessary to configure a particular coupled run.

The namcouple is a text file with the following characteristics:

The first part of namcouple is devoted to configuration of general parameters such as the total run time or the desired debug level. The second part gathers specific information on each coupling (or I/O) field, e.g. their coupling period, the list of transformations or remapping to be performed by OASIS3-MCT and associated configuring lines (described in more details in chapter 4).

Compared to OASIS3, several namcouple inputs have been deprecated in OASIS3-MCT (i.e. they will not be read and not be used) but, for backwards compatibility, they can still appear in the namcouple. These inputs will be noted “UNUSED” in the following text and will not be fully described.

In the next sections, a namcouple example is given and all configuring parameters are described. Additional lines containing different parameters for each transformation are described in chapter 4. A realistic namcouple can be found in oasis3-mct/examples/ sub-directories.

3.1 An example of a simple namcouple

The following simple namcouple configures a run into which an ocean and an atmosphere components are coupled. The ocean running on grid toce provides only the SOSSTSST field to the atmosphere (grid atmo), which in return provides the field CONSFTOT to the ocean. The field CONSFTOT is also explicitly written out to a file flda3.nc and one field SOALBEDO is read from a file.

########## First section #############################################
 $NFIELDS
    4
#
 $RUNTIME
    432000
#
 $NLOGPRT
   2     1    0
#
 $NCDFTYP
   cdf1
#
 $NUNITNO
   901     920
#
 $NMAPDEC
   decomp_wghtfile
#
 $NMATXRD
   ceg
#
 $NWGTOPT
   ignore_bad_index
#
 $SEQMODE
 $CHANNEL
 $JOBNAME
 $NBMODEL
 $INIDATE
 $MODINFO
 $CALTYPE
#
########## Second section #############################################
#
 $STRINGS
#
# Field 1
 SOSSTSST SISUTESU 1 86400  5  sstoc.nc  EXPOUT
 182  149  128  64  toce  atmo   LAG=+14400  SEQ=+1
 P 2 P 0
 LOCTRANS CHECKIN MAPPING  BLASNEW CHECKOUT
#
  AVERAGE
  INT=1
  map_toce_atmo_120315.nc src opt
  1.0  1
  CONSTANT     273.15
  INT=1
#
# Field 2
 CONSFTOT SOHEFLDO 6 86400  4   flxat.nc  EXPORTED
 atmo   toce  LAG=+14400  SEQ=+2
 P 0 P 2
 LOCTRANS  CHECKIN  SCRIPR CHECKOUT
#
  ACCUMUL
  INT=1
  BILINEAR LR SCALAR LATLON 1
  INT=1
#
# Field 3
 CONSFTOT  CONSFTOT 1  86400   1  flda3.nc  OUTPUT
 128  64  128  64 atmo   atmo 
 LOCTRANS
 AVERAGE
#
# Field 4
 SOALBEDO SOALBEDO  17  86400  0  SOALBEDO.nc  INPUT

3.2 First section of namcouple file

The first section of namcouple uses some predefined keywords prefixed by the $ sign to locate the related information. The $ sign must be the first non-blank character on the line but can be in any column. 8 keywords are used by OASIS3-MCT and 5 of these are optional :

3.3 Second section of namcouple file

The second part of the namcouple, starting after the keyword $STRINGS, contains coupling information for each coupling (or I/O) field. Its format depends on the field status given by the last entry on the field first line (EXPORTED, IGNOUT or INPUT in the example above). The field may be9 :

3.3.1 Second section of namcouple for EXPORTED and EXPOUT fields

The first 4 lines for fields with status EXPORTED and EXPOUT are as follows:

   SOSSTSST SISUTESU 1 86400  3  sstoc.nc  EXPOUT
   182  149    128  64  toce  atmo   LAG=+14400 SEQ=+1
   P 2 P 0
   CHECKIN   YAC   CHECKOUT

where the different entries are:

Support to group multiple fields in a single communication

With OASIS3-MCT, it is possible to group multiple fields in a single communication. To activate this option, the user must list the related fields on a single entry line (with a maximum of 5000 characters on one line) through a colon delimited list in the namcouple, for example:

ATMTAUX:ATMTAUY:ATMHFLUX TAUX:TAUY:HEATFLUX 1 3600 3 rstrt.nc EXPORTED

All fields in this group will then use the same namcouple settings (source and target grids, transformations, etc.) for that entry. In the component model codes, these fields are still apparently sent or received one at a time through individual oasis_put and oasis_get. Inside OASIS3-MCT, the fields in this group are stored and a single mapping and send or receive instruction is executed for all fields. This is useful in cases where multiple fields have the same coupling transformations to reduce communication costs by aggregating multiple fields into a single communication.

This option does not put any constraint on the order of the related oasis_put and oasis_get in the codes. This option is compatible with the special case of 3D unbundled fields and it is particularly well suited for lowering the latency of the communications of such large objects.

As they appear in one single entry line, these fields must share the same coupling restart file but this restart file may contain other fields.

3.3.2 Second section of namcouple for OUTPUT fields

The first 3 lines for fields with status OUTPUT are as follows:

 CONSFTOT  CONSFTOT 1  86400   1  flda3.nc  OUTPUT
 128  64  128  64 atmo   atmo   LAG=+14400  SEQ=+1 
 LOCTRANS

where the different entries are:

3.3.3 Second section of namcouple for INPUT fields

The first and only line for fields with status INPUT is:

  SOALBEDO SOALBEDO  1  86400  0  SOALBEDO.nc  INPUT

where the different entries are:

4 Transformations and interpolations

Different transformations and 2D interpolations are available in OASIS3-MCT to adapt the coupling fields from a source model grid to a target model grid. In the following paragraphs, a description of each transformation, with its corresponding configuration lines that the user has to write in the namcouple file, is given. Features that are now deprecated (non functional) compared to prior versions are noted but not described.

4.1 Time transformations

4.2 The pre-processing transformations

Note that previous transformations REDGLO, INVERT, MASK, EXTRAP, CORRECT are not supported anymore.

4.3 The remapping (or interpolation or regridding)

The 3 options for remapping are MAPPING for using an existing set of weights and addresses, or YAC for using the YAC library to generate remapping weights and addresses, or SCRIPR to use the SCRIP library to generate them. Note that INTERP, MOZAIC, NOINTERP, and FILLING are not supported aymore in OASIS3-MCT.

4.3.1 MAPPING

This keyword is used to specify a MAPPING input file containing the weights and addresses to be read and used for remapping.

MAPPING can be used for higher-order remapping, with up to 5 different sets of weights in the MAPPING file and applied to up to 5 different fields transferred through the oasis_put arguments, if present (see section 2.2.7).

As for the other transformations, different mappings can be specified for the different coupling fields. Grid data files grids.nc, masks.nc, and areas.nc are not needed for MAPPING.

This transformation requires at least one configuring line with one filename and two optional string values:

     $MAPNAME  $MAPLOC  $MAPSTRATEGY

Note that if SCRIPR or YAC (see below) is used to calculate the remapping file, MAPPING can still be listed in the namcouple, for example to use it with a specific $MAPLOC or $MAPSTRATEGY option.

4.3.2 YAC

YAC offers the parallel remapping algorithms for 2D interpolation on the sphere from the YAC coupler . YAC routines are in oasis3-mct/lib/yac and yaxt directories.

An interpolation with YAC consists of one or more interpolation methods that form an interpolation stack prescribed in the namcouple, as described below. The name of the resulting remapping file, containing remapping weights and addresses, is composed with the character string appearing on the first YAC line in the namcouple. When YAC performs a remapping, it first checks if the remapping file exists; if it exists, it reads the weights and addresses from the file; if not, it calculates them and store them in a file with that name. When using an existing remapping file, one has to be sure that it was generated in exactly the same conditions than the ones it is currently used for.

YAC needs the grid data files grids.nc containing the longitudes and latitudes of the source and target grid centers and, if the CONSERV remapping is prescribed, also including the longitudes and latitudes of the source and target grid corners. If a mask is defined for a grid in the auxiliary file masks.nc, it will be taken into account in the interpolation (see section 5.1).

The different types of interpolations available with YAC are described in detail in . Examples are also available in oasis3-mct/pyoasis/examples/11-test-interpolation/
common_data/namcouple_yac_bgnc and namcouple_yac_bggd. These are :

YAC first configuration line in the namcouple is:

 # First configuration line for YAC in the namcouple
     $EDGS $EDGT $NINT $FILENAMEAPPEND $NIO

where :

The following types of interpolations can then be specified in the namcouple, each one having a specific line of parameters described here. Note that for all parameters, the default value, if any, can be activated by setting DEFAULT for that parameter.

4.3.3 SCRIPR

SCRIPR gathers the interpolation techniques offered by Los Alamos National Laboratory SCRIP 1.4 library 13. SCRIPR routines are in oasis3-mct/lib/scrip. See the SCRIP 1.4 documentation in oasis3/doc/SCRIPusers.pdf for more details on the interpolation algorithms.

Since OASIS3-MCT_4.0 release, a hybrid MPI+OpenMP parallel version of the SCRIP library is available. It relies on the MPI parallel layout of the calling model but only uses one MPI process per node. To benefit from the OpenMP parallelisation of the SCRIP, one has, of course, to compile OASIS3-MCT with options activating OpenMP, see also section 6.1. In The number of OpenMP threads per MPI process used by SCRIP is set by a dedicated environment variable OASIS_OMP_NUM_THREADS, which can be different from the default number of threads per MPI process used elsewhere in the application, set by the environment variable OMP_NUM_THREADS ; in practice, OASIS_OMP_NUM_THREADS has to be smaller or equal to OMP_NUM_THREADS. For optimum performance, it is recommended to set OASIS_OMP_NUM_THREADS to the number of cores on the node. The regrid_environment directory (see section 6.3.3) in oasis3-mct/examples/regrid_environment gives a practical example on how to use the SCRIP library (or ESMF or XIOS) in parallel to calculate regridding weight-and-address files, see section 6.3.3. The details of the SCRIP parallelisation can be found in and .

When the SCRIP library performs a remapping, it first checks if the file containing the corresponding remapping weights and addresses exists; if it exists, it reads them from the file; if not, it calculates them and store them in a file. In the later case, SCRIP needs the grid data files grids.nc and masks.nc (see section 5.1).

The weight-and-address file is created in the working directory and is by default called
rmp_src_to_tgt_INTTYPE_NORMAOPT.nc, rmp_src_to_tgt_INTTYPE_NBR.nc,
or rmp_src_to_tgt_INTTYPE.nc, where:

One has to take care that the remapping file will have the same name even if other details, like the grid masks or the $MAPLOC or $MAPSTRATEGY options, are changed. When reusing a remapping file, one has to be sure that it was generated in exactly the same conditions than the ones it is used for.

The following types of interpolations are available:

Precautions related to the use of the SCRIPR/CONSERV remapping (consider using YAC remapping instead)

  1. Lambert projection above $NTHRESH or below $STHRESH

    For the 1st order conservative remapping, the weight of a source cell is proportional to area of the source cell intersected by target cell. Using the divergence theorem, the SCRIP library evaluates this area with the line integral along the cell borders enclosing the area. As the real shape of the borders is not known (only the location of the corners of each cell is known), the library assumes that the borders are linear in latitude and longitude between two corners. This assumption becomes less and less valid closer to the pole.

    To correct this inaccuracy, for latitudes above $STHRESH or below $STHRESH, the library evaluates the intersection between two cell borders using a Lambert equivalent azimuthal projection. analyses the impact of the Lambert projection for specific grids.

  2. Another limitation of the SCRIP 1st order conservative remapping algorithm is the assumption, for line integral calculation, that \(sin(latitude)\) is linear in longitude on the cell borders which again is in general not valid close to the pole.

  3. For a proper conservative remapping, the corners of a cell have to coincide with the corners of its neighbour cell, with no “holes” between the cells.

  4. Overlapping cells : If two cells of one same grid overlap, the one with the greater numerical index must be masked in masks.nc for a proper conservative remapping. For example, if the grid cells with i=1 overlaps the grid cells with i=imax, the latter must be masked. If none of the overlapping cells is masked (given the original mask defined in masks.nc), OASIS3-MCT must be compiled with the CPP key TREAT_OVERLAY which will ensure that these rules are respected.

  5. Remapped field values for masked (non-active) target grid cells are set to 0.

  6. If a target grid cell intersects only masked source cells, or falls outside the source grid domain, it will get a zero value unless the DESTNNEI, DESTNNTR, FRACNNEI, or FRACNNTR nor- malisation options are used, in which case it will get the source nearest non-masked neighbour value.

The “True Area” (TR) correction: DESTARTR, DESTNNTR, FRACARTR, or FRACNNTR

The approximations and hypotheses adopted by SCRIP impact its estimation of the grid cell areas. Therefore, to have an exact conservation of the field surface-integrated values, a correction based on the "TRue" (TR) area of the cells can be applied by choosing DESTARTR, DESTNNTR, FRACARTR or FRACNNTR options. These are based respectively on DESTAREA, DESTNNEI, FRACAREA and FRACNNEI normalisations adding the so-called “TR correction”. The true area of the cell, i.e. the one considered by the component model itself, must be provided in the auxiliary file areas.nc in square radians. Equations from (eqn. (27) in particular) are implemented.

Special care is taken in the implementation for “polar" cells in the SCRIP sense. The SCRIP detects cells encompassing a pole or cells with a side going through a pole as “polar" cells and a specific treatment is applied for those cells modifying its area. The resulting estimated area serves as a normalisation factor but its value is not representative of the surface of the cell anymore. For this reason, the TR correction is not activated for those “polar” cells.

More details can be found in section 6 of . A full validation of the TR correction can be found in that report. It is noted there that the TR correction always improves the results even if, in the cases tested, the misfit between the true area and the area evaluated by the SCRIP is always small (except for “polar” cells of course).

Support of vector fields with the SCRIPR remappings

Vector mapping is not supported and will not be supported by OASIS3-MCT. For proper treatment of vector fields, the source code has to send the 3 components of the vector projected in a Cartesian coordinate system as separate fields. The target code has to receive the 3 interpolated Cartesian components and recombine them to get the proper vector field.

4.4 The post-processing stage

Note that SUBGRID, MASKP, REVERSE, GLORED are not supported anymore in OASIS3-MCT.

5 OASIS3-MCT auxiliary data files

OASIS3-MCT uses auxiliary data files, defining the grids of the models being coupled, or containing coupling field restart values or input data values or remapping weights and addresses. These files can be created by the user before the run or by OASIS3-MCT itself in the initialisation phase.

5.1 Grid data files

OASIS3-MCT requires grid data files grids.nc, masks.nc and areas.nc for certain operations:

These NetCDF files can be created by the user before the run or can be written directly at run time by the component model processes in the initialisation phase using the grid data definition routines (see section 2.2.4).

The arrays containing the grid information are dimensioned (nx, ny), where nx and ny are the grid first and second dimension. Unstructured grids or other grids expressed with 1D vectors are supported by setting nx to the total number of grid points and ny to 1.

  1. grids.nc: contains the model grid longitudes and latitudes in double precision REAL arrays. The array names must be composed of a prefix (4 characters), given by the user in the namcouple on the second line of each field (see section 3.3), and of a suffix,.lon or .lat, for respectively the grid point longitudes or latitudes.

    If the YAC/CONSERV or SCRIPR/CONSERV remapping is specified, longitudes and latitudes for the source and target grid corners must also be available in the grids.nc file as double precision REAL arrays dimensioned (nx,ny,nc) where nc is is the maximum number of corners (in the counterclockwize sense, starting by any corner) over all cells; nc can be any number. For cells that do not have the maximum number of distinct corners, we recommend to repeat the last corner as many times as needed to describe nc corners. The names of the arrays must be composed of the grid prefix and the suffix .clo or .cla for respectively the grid corner longitudes or latitudes.

    Longitudes must be given in degrees East in the interval -360.0 to 720.0. Latitudes must be given in degrees North in the interval -90.0 to 90.0. Note that if some grid points overlap, it is recommended to define those points with the same number (e.g. 360.0 for both, not 450.0 for one and 90.0 for the other) to ensure automatic detection of overlap by OASIS3-MCT.

    The corners of a cell cannot be defined modulo 360 degrees. For example, a cell located over Greenwich will have to be defined with corners at -1.0 deg and 1.0 deg but not with corners at 359.0 deg and 1.0 deg.

    Cells larger than 180.0 degrees in longitude are not supported.

    Longitudes and latitudes of grid points or corners can be defined through the oasis_write_grid or oasis_write_corner API routines respectively, see section 2.2.4.

  2. masks.nc: contains the masks for all component model grids in INTEGER arrays. Be careful to follow the historical OASIS3-MCT convention: 0 = not masked (i.e. active), 1 = masked (i.e. not active) for each grid point. The array names must be composed of the grid prefix and the suffix “.msk”. Mask can be defined through the oasis_write_mask API routine, see section 2.2.4.

    File masks.nc may also contain the grid cell static fractions that defines the active (i.e. not masked) part of the cells. Note that those static fractions must not be confused with dynamic fractions supported by the fracwgt option, see section 2.2.7.1. The static fraction fields, if present, are only used in the global CONSERV, CHECKIN and CHECKOUT operations. If both a mask and fractions are defined for a grid, they must be consistent; OASIS3-MCT will abort if they are not coherent or if both are missing. Note that given OASIS3-MCT convention for the mask, a gridcell with mask=0 (active) should have a fractions greater than 0 and a gridcell with mask=1 (inactive) should have a fractions equal to 0. The fraction array name must be composed of the grid prefix and the suffix .frc. Fractions can be defined through the oasis_write_frac API routine, see section 2.2.4.

  3. areas.nc: this file contains mesh surfaces for the component model grids in double precision REAL arrays. The array names must be composed of the grid prefix and the suffix .srf. The surfaces may be given in any units but they must be the same on the source and target sides; furthermore they must be in square radians if the True Area (TR) correction is activated, see section 4.3.

    Surfaces can be defined through the oasis_write_area API routine, see section 2.2.4.

5.2 Coupling restart files

At the beginning of a coupled run, coupling fields having a positive LAG defined in the namcouple are initially read from their coupling restart file on their source grid (see the LAG concept in section 2.5.2). At the end of the run, the coupling restart files are automatically overwritten under the oasis_put when the date+LAG=runtime. It is the user’s responsibility to ensure that the overwritten restart file is copied in the working directory of the next run. The coupling restart files must follow the NetCDF format.

The name of the coupling restart file is given by the 6th character string on the first configuring line for each field in the namcouple and the restart field in that file must have the symbolic name of the coupling field in the namcouple, (see section 3.3). If the coupling exchange involves higher-order fields (see section 2.2.7), the higher-order restart fields in the coupling restart file must have the same name than the coupling field itself prefixed with either av2_ , av3_ , av4_ , or av5_. For example, higher-order restart fields for the SOSSTSST field in section 3.3.1, if needed, would be named av2_SOSSTSST, av3_SOSSTSST, av4_SOSSTSST, and av5_SOSSTSST. These fields have to be provided on the source grid in single or double precision REAL arrays and, as the grid data files, must be dimensioned (nx, ny), where nx and ny are the grid first and second dimension (see section 5.1 above). The shape and indexing of each restart field (and of the corresponding coupling fields exchanged during the simulation) must be coherent with the shape of its grid data arrays.

If the fracwgt option is activated for a field with a LAG (see section 3.3), special care have to be taken on the content of the coupling restart files. If no LOCTRANS ACCUMUL or AVERAGE time transformation is specified for the field, the coupling restart file may contain only the restart field, with its symbolic name, and a dynamic fraction field (see the fracwgt argument of the oasis_put call in section 2.2.7.1) named av1mxxxxxx_YYYYYYYY, where xxxxxx is the 6-digit number of the coupling field and YYYYYYYY is the field symbolic name. But if a LOCTRANS ACCUMUL or AVERAGE time transformation is specified for the field, then the coupling restart files must contain in addition the restart coupling field remapped on the destination grid taking into account the dynamic fraction, named avfwxxxxxx_YYYYYYYY, where xxxxxx is the 6-digit number of the coupling field and YYYYYYYY is the field symbolic name.

Coupling fields coming from different models cannot be in the same coupling restart files, but for each model, there can be an arbitrary number of fields written in one coupling restart file. One exception is when a coupling field sent by a source component model is associated with more than one target field and model; in that case, if coupling restart files are required, it is mandatory to specify different files for the different fields.

If the CPP key LOCTRANS_RESTARTS is activated when compiling OASIS3-MCT, the coupling restart files are also used automatically by OASIS3-MCT to allow partial LOCTRANS time transformation to be saved at the end of a run for exact restart at the start of the next run. Although we do not recommend it as it may have unexpected side effects, this allows a run to end in the middle of a coupling period. In that case, the initial coupling restart file should not contain any LOCTRANS restart fields. For the following runs, it is mandatory that the coupling restart file contains LOCTRANS restart fields coherent with the current namcouple entries. For example, it will not be possible to restart a run with a multiple field entry in the namcouple with a coupling restart file created by a run not activating this multiple file option. See section 4.1 for more detail on the LOCTRANS restart fields in the coupling restart files.

5.3 Input data files

Fields with status INPUT in the namcouple are, at runtime, simply be read in from a NetCDF input file by the target model below the oasis_get call, at appropriate times corresponding to the input period indicated by the user in the namcouple.

The name of the file must be the one given on the first configuring line for that field in the namcouple (see section 3.3.3). There must be one input file per INPUT field, containing a time sequence of the field in a single or double precision REAL array named with the same symbolic name as in the namcouple and dimensioned (nx,ny,time). The time variable has to be an array time(time) expressed in “seconds since beginning of run” (or any other time units as long as the same are used in all components and in the namcouple) . The “time” dimension has to be the unlimited dimension.

5.4 Transformation auxiliary data files

The mapping files to be used for the MAPPING option must be consistent with the files generated by the SCRIP or YAC library to be used for the SCRIPR or YAC transformations (see also section 4.3). The NetCDF files contain the following fields:

dimensions:
      src_grid_size = xxx ;
      dst_grid_size = xxx ;
      num_links = xxx ;
      num_wgts = xxx ;
variables:
      int src_address(num_links) 
      int dst_address(num_links) 
      double remap_matrix(num_links, num_wgts)

where

6 Compiling, running, debugging, load balancing

6.1 Compiling OASIS3-MCT

OASIS3-MCT is a mixed MPI-OpenMP parallel code. Compiling OASIS3-MCT libraries can be done in the oasis3-MCT/util/make_dir directory with the makefile TopMakefileOasis3. In the new OASIS3-MCT v6 compiling environment, two environment variables $(OASIS_COUPLE) and $(OASIS_ENV) have to be defined:

TopMakefileOasis3 includes make.inc which includes the header Makefile make.$(OASIS_ENV) specific to the hardware and compiling environment used. We recommend to compose $(OASIS_ENV) as (computer_name)_(compiler_version)_(MPI_library_version). A run-command file named setup_env_$(OASIS_ENV).sh should also exist in the compiling directory. It defines the computer environment for compiling and running coupled model, in particular the test examples from oasis3-mct/examples, and should to be sourced in the user environment.

Several make.$(OASIS_ENV) header files and setup_env_$(OASIS_ENV).sh run-command files are distributed with the release and can be used as a template to create a custom file for your machine. In those header files, appropriate options must be activated for compiling OASIS3-MCT in debug mode or with OpenMP, when applicable (see respectively the environment variables $(OASIS_DEBUG) and $(OASIS_OPENMP) in those files). Note that is mandatory that the coupled model components, and python modules where applicable, be compiled with the same compilers and system software than the OASIS3-MCT library itself.

After successful compilation, resulting libraries are found in the directory in $(ARCHDIR)/lib16 while object and module files are found in $(ARCHDIR)/build-static and $(ARCHDIR)/build-shared.

OASIS3-MCT has historically created static libraries for use in Fortran source codes. However, C language bindings are now available and python codes are now fully supported. Therefore, the Makefile TopMakefileOasis3 supports compilation of both static and shared libraries.

TopMakefileOasis3 has several targets including:

Log and error messages from compilation are normally saved in the directory /util/make_dir in the files COMP.log and COMP.err or similar. The TopMakefileOasis3 output will direct users to the compile output files.

To interface a component code with OASIS3-MCT and use the module mod_oasis (see section 2.2.1), it is required to include OASIS3-MCT modules from $(ARCHDIR)/include and link with appropriate libraries in $(ARCHDIR)/lib during the compilation and linking.

6.2 CPP keys

The following CPP keys can be used when compiling OASIS3-MCT, in general to be specified in the CPPDEF variable in the header Makefile :

6.3 Examples on how to run OASIS3-MCT

The following examples of running environments are provided in the directory oasis3-mct/examples.

6.3.1 tutorial_communication

The directory oasis3-mct/examples/tutorial_communication contains the files of a tutorial on how to instrument a code with calls to the OASIS3-MCT library in order to couple it with other components. The tutorial involves two toy model codes, ocean.F90 and atmos.F90, which give you a practical example of the use of the OASIS3-MCT API (Application Program Interface) routines. Toy models are skeleton programs that do not contain any real physics or dynamics but that can reproduce real exchanges of coupling fields.

All information about this tutorial is provided in the document tutorial_communication.pdf therein.

6.3.2 spoc

This directory contains the sources used in the Short Online Private Course (SPOC) on “Code Coupling with OASIS3-MCT” developed in the framework of the ESiWACE Centre of Excellence. This SPOC is composed of videos, quizzes and hands-on. The goal is to instrument two toy models to set-up a real coupled model exchanging coupling fields (directory /spoc_communication) and to learn more about OASIS3-MCT regridding functionality (directory /spoc_regridding). If you are interested in attending the SPOC, please visit the online training section of CERFACS web site at https://cerfacs.fr/en/online-traning-sessions/ . Videos and quizzes extracted from the SPOC are also available as Open Education Resources (OER) material at https://www.oercommons.org/courseware/lesson/85340 .

6.3.3 regrid_environment

The regrid_environment directory offers a scripting environment to calculate the regridding weights and the regridding error for specific couple of grids and specific regridding algorithms with either the SCRIP library, ESMF or XIOS. The document regrid_environment_documentation.pdf therein contains all instructions on how to run this tutorial.

6.3.4 Fortran, C and python equivalent (pyoasis/examples)

Different examples implementing the Fortran, C and python APIs are provided as practical illustrations in directory /pyoasis/examples :

The different examples can be launched with the Makefile from directory /pyoasis using targets examples, examples_f or examples_c to run respectively python, Fortran and C examples.

6.4 Debugging

6.4.1 Debug files

If you experience problems while running your coupled model, you can obtain more information in OASIS3-MCT debug files, by increasing the value of the first number on the line below $NLOGPRT in your namcouple, see section 3.2 for details.

6.4.2 Time statistics files

The level of time statistics written out in files named .timers_xxxx depends on the value of the second number on the line below the $NLOGPRT keyword in the namcouple :

The time given for each timer is calculated by the difference between calls to oasis_timer_start() and oasis_timer_stop() and is the accumulated time over the entire run. Here is an overview of the meaning of the different timers implemented by default. 17

6.5 Load balancing analysis of coupled model components

An efficient use of the allocated computing resources in a coupled system requires the harmonisation of the component execution speed. This operation, called load balancing, is often neglected, either because of the apparent resource abundance or practical difficulties. To facilitate this work, a load balancing analysis functionality is included in OASIS3-MCT and can be activated by setting to 1 the third number under $NLOGPRT in the namcouple configuration file (see section 3.2). Some details on this functionality are provided here and more information can be found in the balancing_documentation.pdf file in oasis3-mct/util/load_balancing directory.

When activated, the load balancing analysis functionality provides general computing information (simulation time, speed, waiting time, etc.) in a text file load_balancing_info.txt for the coupled model and for each component; this global information can help to allocate resources in a balanced way.

This functionality also outputs the full timeline of all OASIS3-MCT related events, for any of the allocated resources. This timeline is saved in one NetCDF file per coupled component, timeline_XXX_component.nc where XXX is the component name. It provides the comprehensive sequence of all operations related to the coupling (field put and get through MPI, field output on disk, field interpolation and mapping, field reading on disk, restart writing, initialisation and termination phase of the OASIS3-MCT setup) so that any simulation slow down in link with the use of the OASIS3-MCT library can be identified. The analysis of the coupling field exchanges, amongst all coupling events, allows to not only identify the waste of resources by components which are recurrently waiting for their coupling fields but it also reveals other bottlenecks such as disk access or model internal load imbalance. The full picture of these events makes possible an optimal load balancing, even for the most complex configurations.

7 Conservative remapping with the fracwgt option

When a coupling field is valid only on a fraction of the source cells (fractional source field), there are different interpretations about the destination field that should be obtained on the destination grid after conservative remapping.

A destination scaled field, i.e. the remapping of the original field multiplied by its valid fraction on each cell, ensures conservation when the full destination cell area of the target grid is considered. However, because of the source fractions, it can become nonphysically small.

Another way to consider the problem is to calculate a destination normalized scaled field that will be in the physical range of the original field. This destination normalized scaled field will be valid only on a fraction of the target, the active destination surface that can be calculated with the active destination surface coefficient.

The new option fracwgt in OASIS3-MCT v6 allows to pass, in the source component as argument of the oasis_put, the active fraction of the source cells together with the source coupling field and to obtain, in the target component in the oasis_get, the destination normalized scaled field. It also allows to calculate the active destination surface coefficient.

For instantaneous fields, option fracwgt is not essential as the user could get the active destination surface coefficient simply with the conservative remapping of the source fraction field. He/she could then obtain the destination normalized scaled field by applying the element-wise division of the destination scaled field by the active destination surface coefficient.

However, for accumulated or averaged fields, option fracwgt is essential to obtain the destination accumulated normalized scaled field and the accumulated active destination surface coefficients, as they cannot be calculated in the models because of the non permutation of the time and remapping operations.

An example of how to use the fracwgt option is given in /pyoasis/examples/13-test-fracwgt.

7.1 Introduction

Conservative remapping is relevant for physical fields representing a flux or a concentration.
We can think of energy fluxes (\(W/m^2\)), precipitation rates (\(mm/m^2\)), tracer concentrations (ppb or similar).
The remapping is required to:

  1. preserve the total energy or mass

  2. avoid abnormal local values of fluxes or concentrations

In models with no sub-grid fractions, the surface integral quantity to be preserved is the sum of the area of the unmasked source cells times the field on the cell \[\sum_iA_{Si}F_{Si}\] where \(A_{Si}\) and \(F_{Si}\) are respectively the area and the field of the source cell \(i\). A conservative remapping will grant that \[\sum_iA_{Si}F_{Si} = \sum_jA_{Dj}F_{Dj}\] where \(A_{Dj}\) and \(F_{Dj}\), the area and the field of the target cell \(j\), can be expressed as: \[F_{Dj} = \sum_iw_{ij}F_{Si}\] \[A_{Dj} = \sum_i\omega_{ij}A_{Si}\] \(\omega_{ij}\), the fraction of the source cell \(i\) overlapping the destination cell \(j\), is related to the conservative remapping weight \(w_{ij}\) representing the area normalized fraction of the destination cell \(j\) intersected by the source cell \(i\) by, as in ,: \[w_{ij}=\omega_{ij}\frac{A_{Si}}{A_{Dj}}\] Notice that, because of some assumptions on the representation of the cell edges on a sphere and to rounding errors, the \(A_{Dj}\) surface can slightly differ from the surface \(A^d_{Dj}\) prescribed in the destination model, but we will neglect this difference here.

7.2 What is a fractional field?

For some models, a process represented on the model grid actually takes place only on a subset of it, whose boundaries are described at a finer resolution than the grid. The numerical translation of this choice is an extra mask-like field, providing the active (or unmasked) fraction \(f_i\in[0,1]\) for every cell.
The field is therefore described with values ranging in the usual physical valid interval (cf. point 2. in the introduction), yet the integral quantity is scaled down to \[\sum_iA_{i}f_iF_{i}\] This scaling can be interpreted in two ways

  1. On cells with usual area \(A_i\), the flux is lowered to the scaled field \(F^f_i=f_iF_i\)

  2. The flux \(F_i\), with values in the usual physical range, is active on a smaller active source surface \(A^a_i = A_if_i\)

7.3 Instantaneous remapping

When applying the conservative remapping on an instantaneous fractional source field (i.e. in OASIS jargon, no LOCTRANS or LOCTRANS/INSTANT time transformation), we grant the conservation of the scaled integral source quantity \[\sum_iA_{Si}f_{Si}F_{Si}\] Once again, we can use different interpretations.
If we consider that we are interpolating a scaled field \(F^f_{Si}=f_{Si}F_{Si}\) with a standard conservative remapping, and we are receiving the resulting destination scaled field \[\begin{equation} \label{eqn:inst_scaled} F^f_{Dj} = \sum_iw_{ij}f_{Si}F_{Si} \end{equation}\] then we will grant, as defined by the conservative remapping \[\sum_iA_{Si}f_{Si}F_{Si} = \sum_jA_{Dj}F^f_{Dj}\] where \(A_{Dj}\) is the full destination cell area. Yet the local values \(F^f_{Dj}\) could locally become nonphysically small.
It is now essential to define the active destination surface coefficients, i.e. the active fraction of every destination cell, \(f_{Dj}\), with the conservative remapping of the source grid fractions: \[\begin{equation} \label{eqn:inst_dest_fract} f_{Dj} = \sum_i{w_{ij}f_{Si}} \end{equation}\] Using this coefficient \(f_{Dj}\), we can define a destination normalized scaled field that will be in the physical range of the original field as \[\begin{equation} \label{eqn:normalized_field} F^{fn}_{Dj} = \frac{F^f_{Dj}}{f_{Dj}} \end{equation}\] Note that if the active fraction of all source cells associated to one target cell \(f_{Si}\) are zero then both \(F^{f}_{Dj}\) and \(f_{Dj}\) are zero and equation [eqn:normalized_field] becomes undefined. In that case, in OASIS3-MCT v6, we set \(F^{fn}_{Dj}\) equal to zero.

In order to preserve the integral quantity we have to scale back the field: \[\sum_iA_{Si}f_{Si}F_{Si} = \sum_jA_{Dj}f_{Dj}F^{fn}_{Dj}\] For computing a conservative integral quantity on the destination side, the active destination surface coefficient \(f_{Dj}\) can be used to define the active destination surface : \[\begin{equation} \label{eqn:active_dest_area} A^a_{Dj} = A_{Dj}f_{Dj} \end{equation}\] so that \[\sum_iA_{Si}f_{Si}F_{Si} = \sum_iA^a_{Si}F_{Si} =\sum_jA^a_{Dj}F^{fn}_{Dj}\] Since OASIS3-MCT v6, the oasis_put routine comes with an extra optional argument fracwgt. If the argument is present, it has to correspond to the active source fraction field \(f_S\), and the matching oasis_get will receive the normalized scaled field \(F^{fn}_D\).
Notice that for technical reasons, the active destination surface coefficient \(f_D\) cannot be recovered as an extra output of the oasis_get call. The user can recover it in two ways:

  1. send the active source fraction field \(f_S\) as a separate coupling field through a standard conservative remapping and directly get \(f_D\) (equation [eqn:inst_dest_fract]) ; or

  2. compute the ratio between the destination scaled field and the destination normalized scaled field

    1. in the source model compute the scaled source field \(F^f_S = f_SF_S\)

    2. send this scaled field with a standard oasis_put call (i.e. without a fracwgt argument) and on the destination side store the corresponding (non normalized) scaled field \(F^f_D\)

    3. send the (non scaled) source field \(F_S\) passing fracwgt=\(f_S\) to the oasis_put call and on the destination side store the corresponding normalized scaled field \(F^{fn}_D\) (equation [eqn:normalized_field])

    4. compute the active destination surface coefficient as an element-wise division \(f_D = F^f_D \odot \frac{1}{F^{fn}_D}\)

Once \(f_D\) is computed, the active destination surface to be used for computing integral quantities is finally obtained by scaling the destination cells surfaces \(A^a_D = A_D \odot f_D\)

Notice that it is mandatory that the treatments described in the namcouple for the two remappings b. and c. are strictly the same.

Even if the implementation of the first method is easier for instantaneous fields, we suggest using the second one that relies on the fracwgt option because, as we will show in the following sections, it is compatible with time accumulated or time averaged remapping.

7.4 Time accumulation

Let’s now consider the case in which the destination field has to be computed as the accumulation of remapped source fields at different times (i.e. in OASIS jargon, LOCTRANS/ACCUMUL time transformation) \[\begin{equation} \label{destination_accumulatedblasoldcoeff} F^+_{Dj}=\sum_t{\alpha F^t_{Dj}+\beta}=\sum_t{\alpha\left(\sum_i{w_{ij}F^t_{Si}}\right)+\beta} \end{equation}\]

where \(\alpha\) and \(\beta\) are the coefficient of an affine transform that can be indicated with the BLASOLD method. Because of the linearity of the operators we can neglect them in the following for the sake of readability. We therefore define the destination accumulated field as \[\begin{equation} \label{destination_accumulatedblasold} F^+_{Dj}=\sum_t{F^t_{Dj}}=\sum_t{\sum_i{w_{ij}F^t_{Si}}} \end{equation}\]

In the case where no subgrid fractions are used, the accumulating sum and the linear mapping can permute and we can easily obtain the destination accumulated field as the remapping of the accumulated source fields \[\begin{equation} \label{destination_accumulated} F^+_{Dj}=\sum_t{\sum_i{w_{ij}F^t_{Si}}} = \sum_i{w_{ij}\sum_t{ F^t_{Si}}} \end{equation}\] In that case, we can rewrite the conservative remapping as \[\begin{equation} \sum_t{\sum_i{A_{Si}F^t_{Si}}} = \sum_t{\sum_j{A_{Dj}F^t_{Dj}}} = \sum_jA_{Dj}{\sum_t{F^t_{Dj}}} = \sum_j{A_{Dj}F^+_{Dj}} \end{equation}\] If we introduce a time varying fractional coefficient in the time accumulation equations, we can obtain the corresponding destination accumulated scaled field by sending and remapping \(f^t_{Si}F^t_{Si}\): \[\begin{equation} \label{eqn:nonnorm_accumulate_scaled} F^{+f}_{Dj} = \sum_iw_{ij}\sum_t{f^t_{Si}F^t_{Si}} \end{equation}\] In that case, the conservation equation is satisfied when the full destination area is used \[\begin{equation} \ \sum_t{\sum_i{A_{Si}f^t_{Si}F^t_{Si}}} = \sum_i{A_{Si}\sum_t{f^t_{Si}F^t_{Si}}} = \sum_j{A_{Dj}F^{+f}_{Dj}} \end{equation}\] However, \(F^{+f}_{Dj}\) will not be physically consistent with the source field because it has been scaled by the source fractions. If we want to preserve the physical range of the destination field, we should compute the destination accumulated normalized scaled field that, given ([eqn:inst_scaled]) and ([eqn:inst_dest_fract]) for each timestep, can be written as: \[\begin{equation} \label{eqn:accumulate_scaled} F^{+fn}_{Dj}=\sum_t{\frac{\sum_i{w_{ij}f^t_{Si}F^t_{Si}}}{\sum_i{w_{ij}f^t_{Si}}}}=\sum_t{\frac{F^{ft}_{Dj}}{f^t_{Dj}}} \end{equation}\] Note that, as for equation [eqn:normalized_field], if for any timestep the active fraction of all source cells associated to one target cell \(f^{t}_{Si}\) are zero then both \(F^{ft}_{Dj}\) and \(f^{t}_{Dj}\) are zero and equation [eqn:accumulate_scaled] becomes undefined. In that case, in OASIS3-MCT v6, we set \(F^{+fn}_{Dj}\) equal to zero.

Notice that in equation [eqn:accumulate_scaled], the summation on the source grid cells does not permute with the summation in time and the summation of the \(N_{t}\) time contribution has to be carried on on the destination grid after remapping: it requires \(2N_t\) sparse multiplications by the weight matrix.
Once again we can derive the expression of the active surface for every destination cell \(A^a_{Dj}\) by imposing \[A^a_{Dj}F^{+fn}_{Dj}=A_{Dj}F^{+f}_{Dj}\] hence \[\begin{equation} A^a_{Dj}=A_{Dj}\frac{F^{+f}_{Dj}}{F^{+fn}_{Dj}} \end{equation}\] which corresponds to the definition of accumulated active destination surface coefficients: \[\begin{equation} \label{eqn:equivalent_dest_frac} f^+_{Dj} = \frac{F^{+f}_{Dj}}{F^{+fn}_{Dj}} \end{equation}\] Since the user cannot recover it directly, the computation of \(f^+_D\) requires the following steps:

  1. at every put time, in the source model compute the scaled source field \(F^f_S = f_SF_S\) and send this scaled field through a standard oasis_put call (i.e. without a fracwgt argument) ;

  2. send the (non scaled) source field \(F_S\) passing fracwgt=\(f_S\) to the oasis_put call;

  3. on the destination side receive and store the (non normalized) accumulated scaled field \(F^{+f}_D\) (equation [eqn:nonnorm_accumulate_scaled]) corresponding to the fields sent in 1 ;

  4. on the destination side receive and store the accumulated normalized scaled field \(F^{+fn}_D\) (equation [eqn:accumulate_scaled]) corresponding to the fields sent in 2 ;

  5. compute the accumulated active destination surface coefficients as an element-wise division \(f^{+}_D = F^{+f}_D \odot \frac{1}{F^{+fn}_D}\) (equation [eqn:equivalent_dest_frac]).

Notice that it is mandatory that the treatments described in the namcouple for the two remappings 1.\(\rightarrow\)3. and 2.\(\rightarrow\)4. are strictly the same.

7.5 Time average

In the case of a time average with individual contributions at equal intervals, (i.e. in OASIS jargon, LOCTRANS/AVERAGE time transformation) the average is obtained by dividing the time accumulated fields by the number of contributions \(N_t\). It’s easy to see that the coefficient \(1/N_t\) can permute with all the time summations and that the active fraction on the destination grid is the same as in the time accumulation case.

\[\overline{f}_{Dj} = \frac{\overline{F}^{f}_{Dj}}{\overline{F}^{fn}_{Dj}} =\frac{F^{+f}_{Dj}/N_t}{F^{+fn}_{Dj}/N_t} = \frac{F^{+f}_{Dj}}{F^{+fn}_{Dj}} = f^+_{Dj}\] The case with unevenly spaced time contributions would be more complicated, but it is not implemented in OASIS3-MCT v6.

7.6 An example

Let’s consider a destination cell with total surface of 5 sq. mt. intersecting four source cells by, respectively, 1, 0.5, 1.5 and 2 sq. mt as in Fig.7.1

Source and destination grids for the test case

The conservative remapping weights are therefore:
\(w_1 = 0.2\); \(w_2 = 0.1\); \(w_3 = 0.3\); \(w_4 = 0.4\)
At time 1, the active source surface covers the following fractions:
\(f^{1}_{S1} = 1.0\); \(f^{1}_{S2} = 0.5\); \(f^{1}_{S3} = 0.25\); \(f^{1}_{S4} = 0.5\)
and the source fields is:
\(F^{1}_{S1} = 1.0\); \(F^{1}_{S2} = 3.0\); \(F^{1}_{S3} = 4.0\); \(F^{1}_{S4} = 2.0\)
If we neglected the sub-grid fractions, the remapped field at time 1 would be \[F^{1}_{Dj}=\sum_i{w_iF^1_{Si}}=2.5\] and it’s contribution to the integral quantity on the destination grid would be \[A_{Dj}F^{1}_{Dj}=12.5\] If we take into consideration the source fractions, the destination scaled field as in eqn. ([eqn:inst_scaled]) is \[F^{1f}_{Dj} = \sum_iw_{i}f^1_{Si}F^1_{Si} = 1.05\] It’s contribution to the integral quantity on the destination grid is reduced to: \[A_{Dj}F^{1f}_{Dj} = 5.25\] Taking into consideration the active destination surface coefficients as in eqn. ([eqn:inst_dest_fract]) \[f^1_{Dj} = \sum_i{w_{i}f^1_{Si} = 0.525}\] We can get the destination normalized scaled field as in eqn. ([eqn:normalized_field]) \[F^{1fn}_{Dj} = \frac{F^{1f}_{Dj}}{f^1_{Dj}} = 2.0\] and the active destination surface as in eqn. ([eqn:active_dest_area]) \[A^{1a}_{Dj} = A_{Dj}f^1_{Dj} = 2.625\] and find again the appropriate contribution to the integral quantity on the destination grid \[A^{1a}_{Dj}F^{1fn}_{Dj} = 5.25\]
Let’s now suppose that at time 2, the active source surface covers the following fractions:
\(f^{2}_{S1} = 1.0\); \(f^{2}_{S2} = 0.25\); \(f^{2}_{S3} = 0.125\); \(f^{2}_{S4} = 0.25\)
and the source fields is:
\(F^{2}_{S1} = 10.0\); \(F^{2}_{S2} = 30.0\); \(F^{2}_{S3} = 40.0\); \(F^{2}_{S4} = 20.0\)

The instantaneous destination scaled field will then be: \[F^{2f}_{Dj} = \sum_iw_{i}f^2_{Si}F^2_{Si} = 6.25\] It’s contribution to the integral quantity: \[A_{Dj}F^{2f}_{Dj} = 31.25\] The active destination surface coefficient: \[f^2_{Dj} = \sum_i{w_{i}f^2_{Si} = 0.3625}\] The destination normalized scaled field: \[F^{2fn}_{Dj} = \frac{F^{2f}_{Dj}}{f^2_{Dj}} = 17.2414\] and the corresponding active destination surface : \[A^{2a}_{Dj} = A_{Dj}f^2_{Dj} = 1.8125\] which confirms the appropriate contribution: \[A^{2a}_{Dj}F^{2fn}_{Dj} = 31.25\]
If we accumulate the field on the two time steps, neglecting the sub-grid fractions we obtain
\(F^{+}_{S1} = 11.0\); \(F^{+}_{S2} = 33.0\); \(F^{+}_{S3} = 44.0\); \(F^{+}_{S4} = 22.0\)
and the destination accumulated field would be (equation [destination_accumulated]) \[F^{+}_{Dj}=\sum_i{w_iF^{+}_{Si}}=27.5\] Meanwhile, if we accumulate the scaled source fields we have
\(F^{+f}_{S1} = 11.0\); \(F^{+f}_{S2} = 9.0\); \(F^{+f}_{S3} = 6.0\); \(F^{+f}_{S4} = 6.0\)
and the destination accumulated scaled field (equation [eqn:nonnorm_accumulate_scaled]) has a much lower value: \[F^{+f}_{Dj}=\sum_i{w_iF^{+f}_{Si}}=7.3\] It’s contribution to the integral quantity on the destination grid is: \[A_{Dj}F^{+f}_{Dj} = 36.5\]
We can obtain a more representative value by accumulating the instantaneous normalized scaled fields as in eqn. ([eqn:accumulate_scaled]) \[F^{+fn}_{Dj}=\sum_t{\frac{F^{ft}_{Dj}}{f^t_{Dj}}}=19.24137931\] and derive its equivalent active destination cell fraction from eqn. ([eqn:equivalent_dest_frac]) \[f^+_{Dj} = \frac{F^{+f}_{Dj}}{F^{+fn}_{Dj}} = 0.379390681\] and an active surface: \[A^a_{Dj}=A_{Dj}f^+_{Dj}=1.896953405\]
Since the average values are just the half sum of the instantaneous values, the fractions are the same as in the accumulation case.
The averaged scaled source fields on the four cells are
\(\overline{F}^{f}_{S1} = 5.5\); \(\overline{F}^{f}_{S2} = 4.5\); \(\overline{F}^{f}_{S3} = 3.0\); \(\overline{F}^{f}_{S4} = 3.0\)
The scaled remapped field is: \[\overline{F}^{f}_{Dj}=\sum_i{w_i\overline{F}^{f}_{Si}}=3.65\] It’s contribution to the integral quantity on the destination grid is: \[A_{Dj}\overline{F}^{f}_{Dj} = 18.25\]
By averaging the normalized remapped values we obtain the more representative averaged field \[\overline{F}^{fn}_{Dj}=9.620689655\] and derive its equivalent active destination cell fraction \[\overline{f}_{Dj} = \frac{\overline{F}^{f}_{Dj}}{\overline{F}^{fn}_{Dj}} = 0.379390681\] which is the same as in the accumulation case and implies an active surface: \[A^a_{Dj}=A_{Dj}\overline{f}_{Dj}=1.896953405\]

8 Changes between the different versions of OASIS3-MCT

The evolution between the different versions of OASIS3-MCT can be followed in real-time by registering on the Redmine project management site at https://inle.cerfacs.fr/ (see "Register" at the right top of the page). On this site, registered users can browse the sources and consult tickets describing bug fixes and developments. To follow day to day evolution of the OASIS3-MCT sources, it is also possible to have your e-mail added to the list of addresses to which the log files of the SVN checkins are automatically sent; contact oasishelp@cerfacs.fr if you wish to be added to that list.

8.1 Changes between OASIS3-MCT v6 and OASIS3-MCT_5.0

The last version of the coupler, OASIS3-MCT v6, comes with the following new functionalities and improvements :

8.2 Changes between OASIS3-MCT_5.0 and OASIS3-MCT_4.0

The last version of the coupler, OASIS3-MCT_5.0, comes with the following novelties:

8.3 Changes between OASIS3-MCT_4.0 and OASIS3-MCT_3.0

Different developments were realised to improve the parallel performance of OASIS3-MCT_4.0. These developments are detailed in .

The other new features offered by OASIS3-MCT_4.0 are the following:

8.4 Changes between OASIS3-MCT_3.0 and OASIS3-MCT_2.0

The main evolution of OASIS3-MCT_3.0 with respect to OASIS3-MCT_2.0 is the support of coupling exchanges between parallel components deployed in much more diverse configurations than before, for example, within one same executable between components running concurrently on separate sets of tasks or between components running sequentially on overlapping sets of tasks. All details are provided in section 2.1.

This new version also includes:

8.5 Changes between OASIS3-MCT_2.0 and OASIS3-MCT_1.0

The main changes and bug fixes new in OASIS3-MCT_2.0 are the following:

8.6 Changes between OASIS3-MCT_1.0 and OASIS3.3

8.6.1 General architecture

8.6.2 Changes in the coupling interface in the component models

8.6.3 Functionality not offered anymore

8.6.4 New functionality offered

8.6.5 Changes in the configuration file namcouple

8.6.6 Other differences

10

Alfeld, P., Neamtu, M., Schumaker, L. L. Bernstein-Bézier polynomials on spheres and sphere-like surfaces Computer Aided Geometric Design, 1996

https://doi.org/10.1016/0167-8396(95)00030-5

Cassou, C., P. Noyret, E. Sevault, O. Thual, L. Terray, D. Beaucourt, and M. Imbard: Distributed Ocean-Atmosphere Modelling and Sensitivity to the Coupling Flux Precision: the CATHODe Project. , 126, No 4: 1035-1053, 1998.

Chavas, J., E. Audit, L. Coquart, and S. Valcke: Conservative Regridding When Grid Cell Edges Are Unknown - Case of SCRIP , , CECI, Université de Toulouse, CNRS, Toulouse, France

Coquart, L., E. Maisonnave, E. and S. Valcke: Using Open MP in OASIS3-MCT for the N-nearest-neighbor remapping , CECI, Université de Toulouse, CNRS, Toulouse, France

Craig, A., S. Valcke and L. Coquart : Development and performance of a new version of the OASIS coupler, OASIS3-MCT_3.0 , 10: 3297–3308

https://doi.org/10.5194/gmd-10-3297-2017

Craig, A., S. Valcke: OASIS3-MCT4.0 Timing Study with MCT 2.10.beta1 , CECI, Université de Toulouse, CNRS, Toulouse, France

Craig, A. : GSSPOS and BSSPOS options for the global conservation in OASIS3-MCT , CECI, Université de Toulouse, CNRS, Toulouse, France

Guilyardi, E., G. Madec, L. Terray, M. Déqué, M. Pontaud, M. Imbard, D. Stephenson, M.-A.  Filiberti, D. Cariolle, P. Delecluse, and O. Thual. Simulation couplée océan-atmosphère de la variabilité du climat. , t. 320, série IIa:683–690, 1995.

Kara, A. Birol, Alan J. Wallcraft, and Harley E. Hurlburt : A Correction for Land Contamination of Atmospheric Variables near Land?Sea Boundaries , 37.4: 803-818

https://doi.org/10.1175/JPO2984.1

Hanke, M., R Redler, R., T. Holfeld and M. Yastremsky. YAC 1.2.0: new aspects for coupling software in Earth system modelling , 9: 2755-2016

https://doi.org/10.5194/gmd-9-2755-2016

Hanke, M. and R Redler. New features with YAC 1.5.0, DWD report 2019,

https://doi.org/10.5676/DWD_pub/nwv/icon_003

He, Y. and C. H. Q Ding. Using Accurate Arithmetics to Improve Numerical Reproducibility and Stability in Numer- ical Applications. The Journal of Supercomputing, 18, 259

https://doi.org/10.1023/A:1008153532043

, 2001.

Jacob, R., J. Larson, and E. Ong: MxN Communication and Parallel Interpolation in CCSM3 Using the Model Coupling Toolkit. , 19(3), 293-307 2005

Jones, P.: Conservative remapping: First- and second-order conservative remapping. Mon Weather Rev, 127, 2204-2210, 1999.

Jonville, G., A. Piacentini, S. Valcke and M. Hanke: Integration of the YAC regridding library into the OASIS3-MCT coupler , CERFACS, Toulouse, France, 2024.

https://hal.science/hal-05043920v1/file/TR-CMGC-24-182_yac_in_oasis.pdf

Jonville, G. and S. Valcke: Analysis of SCRIP conservative remapping in OASIS3-MCT Part B , CERFACS, Toulouse, France, 2020.

Kritsikis, E., Aechtner, M., Meurdesoif, Y., and Dubos, T., Conservative interpolation between general spherical meshes, , 10, 425?431, 2017

https://doi.org/10.5194/gmd-10-425-2017

Larson, J., R. Jacob, and E. Ong: The Model Coupling Toolkit: A New Fortran90 Toolkit for Building Multiphysics Parallel Coupled Models. , 19(3), 277-292, 2005

Maisonnave, E. Locally conservative OASIS interpolation using target grid nearest neighbours , CERFACS, Toulouse, France, 2020.

Maisonnave, E., L. Coquart, and A. Piacentini: A better diagnostic of the load imbalance in OASIS based coupled systems , CERFACS, Toulouse, France, 2020.

Mirin, A. A., and P. H. Worley : Improving the Per- formance Scalability of the Community Atmosphere Model. , 26, 17-30

Noyret, P., E. Sevault, L. Terray and O. Thual. Ocean-atmosphere coupling. , 1994.

Piacentini, A., E. Maisonnave, G. Jonville, L. Coquart and S. Valcke: A parallel SCRIP interpolation library for OASIS, , CERFACS, Toulouse, France, 2018.

Piacentini, A. and E. Maisonnave: Interactive visualisation of OASIS coupled models load imbalance , CERFACS, Toulouse, France, 2020.

Pontaud, M., L. Terray, E. Guilyardi, E. Sevault, D. B. Stephenson, and O. Thual. Coupled ocean-atmosphere modelling - computing and scientific aspects. In 2nd UNAM-CRAY supercomputing conference, Numerical simulations in the environmental and earth sciences Mexico-city, Mexico, 1995.

Sevault, E., P. Noyret, and L. Terray. Clim 1.2 user guide and reference manual. , CERFACS, 1995.

Taylor, K. E. Truly conserving with conservative remapping methods. , 17:415–430, 2024 /gmd-17-415-2024,

https://gmd.copernicus.org/articles/17/415/2024/

Terray, L. and O. Thual. Oasis: le couplage océan-atmosphère. , 10:50–61, 1995.

Terray, L. and O. Thual. Coupled ocean-atmosphere simulations. In High Performance Computing in the Geosciences, proceedings of the Les Houches Workshop F.X. Le Dimet Ed., Kluwer Academic Publishers B.V, 1993.

Terray, L., E. Sevault, E. Guilyardi and O. Thual OASIS 2.0 Ocean Atmosphere Sea Ice Soil User’s Guide and Reference Manual , CERFACS, 1995.

Terray, L. O. Thual, S. Belamari, M. Déqué, P. Dandin, C. Lévy, and P. Delecluse. Climatology and interannual variability simulated by the arpege-opa model. , 11:487–505, 1995

Terray, L., S. Valcke and A. Piacentini: OASIS 2.3 Ocean Atmosphere Sea Ice Soil, User’s Guide and Reference Manual, , CERFACS, Toulouse, France, 1999.

Valcke, S., Piacentini, A. and Jonville, G. Benchmarking of regridding libraries used in Earth System Modelling: SCRIP, YAC, ESMF and XIOS , CERFACS, Toulouse, France, 2021.

Valcke, S. and A. Piacentini Analysis of SCRIP conservative remapping in OASIS3-MCT , CERFACS, Toulouse, France, 2019.

Valcke, S., L. Coquart, A. Craig, G. Jonville, E. Maisonnave, A. Piacentini Multithreaded or thread safe OASIS version including performance optimizations to adapt to many-core architectures, IS-ENES2 deliverable D2.3 , CERFACS, Toulouse, France, 2018.

Valcke, S., G. Jonville, R. Ford, M. Hobson, A. Porter and G. Riley Report on benchmark suite for evaluation of coupling strategies , CERFACS, Toulouse, France, 2018.

http://cerfacs.fr/wp-content/uploads/2017/05/GLOBC-TR-IS-ENES2_D10.3_MAI2017.pdf

Valcke, S., T. Craig, L. Coquart: OASIS3-MCT User Guide, OASIS3-MCT 3.0, , CERFACS, Toulouse, France, 2010.

Valcke, S.: The OASIS3 coupler: a European climate modelling community software , 6:373–388

Valcke, S., T. Craig, L. Coquart: OASIS3-MCT User Guide, OASIS3-MCT 2.0, , CERFACS, Toulouse, France, 2013.

Valcke, S., T. Craig, L. Coquart: OASIS3-MCT User Guide, OASIS3-MCT 1.0, , CERFACS, Toulouse, France, 2012.

Valcke, S., M. Hanke, L. Coquart: OASIS4_1 User Guide , CERFACS, Toulouse, France, 2011.

Valcke, S.: OASIS3 User Guide (prism_2-5) , CERFACS, Toulouse, France, 2006.

Valcke, S.: OASIS4 User Guide (OASIS4_0_2), , CERFACS, Toulouse, France, 2006.

Valcke, S., A. Caubel, R. Vogelsang, and D. Declat: OASIS3 User’s Guide (oasis3_prism_2-4), , CERFACS, Toulouse, France, 2004.

Valcke, S., A. Caubel, D. Declat and L. Terray: OASIS3 Ocean Atmosphere Sea Ice Soil User’s Guide, , CERFACS, Toulouse, France, 2003.

Valcke, S., L. Terray and A. Piacentini: OASIS 2.4 Ocean Atmosphere Sea Ice Soil, User’s Guide and Reference Manual, , CERFACS, Toulouse, France, 2000.

Voldoire, A. River to ocean models interpolation , CNRM, Université de Toulouse, Météo-France, CNRS, Toulouse, France, 2020

  1. A list of coupled models realised with OASIS3-MCT can be found at https://oasis.cerfacs.fr/en/oasis-coupled-models/↩︎

  2. https://web.cels.anl.gov/projects/climate/mct/↩︎

  3. Note also that with releases earlier than OASIS3-MCT_5.2 (2022/11/02), it was not allowed to associate more than one grid to one partition (this could cause problems with the global CONSERV operation.)↩︎

  4. Parameters OASIS_In, OASIS_Out are defined in oasis3-mct/lib/psmile/src/mod_oasis_parameters.F90↩︎

  5. But the field rank does not necesseraly have to match the rank of the grid partition description (i.e. it can be expressed in either 1D or 2D) as long as it provides overall the same number of elements.↩︎

  6. When there is a lag, the first and last instance of the source field in the debug netCDF file (EXPOUT fields, see section 3.3) always correspond respectively to the field read from and written to the restart file.↩︎

  7. In that case, if set, the SEQ indices must be identical. If by mistake, SEQ is set to 1 for fld1 and 2 for fld2, then the coupled model will abort because at runtime, the coupler will detect in model 2 that fld2 was sent before fld1 was received which is out of sequence as defined by the SEQ settings.↩︎

  8. NetCDF will read and/or overwrite existing files of any NetCDF file type, and the file type will remain unchanged in that case.↩︎

  9. The status AUXILARY is not supported anymore.↩︎

  10. For 1D field, put 1 as the second dimension↩︎

  11. For 1D field, put 1 as the second dimension↩︎

  12. This is especially important for cells from regular grids that are located close to the poles.↩︎

  13. See also http://climate.lanl.gov/Software/SCRIP/ and the copyright statement in appendix 1.3.3.↩︎

  14. The only exceptions are for Gaussian Reduced (D) grids for ( BILINEAR, BILINEARNF, BICUBIC and BICUBICNF; in that case, if the Gaussian-reduced grid is stored from North to South the number of bins is the number of latitude circles of the grid (minus one, to be precise), independently of $NBIN; for Gaussian-reduced grid stored from South to North to South, the bin definition will not work and the interpolation will become a 4 distance-weighted nearest-neighbour for all target points.↩︎

  15. In principle, the fractions can be defined for both the source and target grids. But for ocean-atmosphere coupling, we strongly encourage the following best practice for a consistent ocean-atmosphere coupled system. Indeed, to have a well-posed coupled problem, the ocean and the atmospheric total surfaces must be the same allowing global conservation of integrated quantities. To do so, the original ocean mask should be taken as it is from the ocean model. For the atmosphere, cell fractions should be defined by the conservative remapping of [1 - ocean mask] on the atmospheric grid, retaining fractions above a certain threshold, to be fixed by the user. These atmospheric cell fractions should be used in the atmospheric model to define the % of ocean (water) subsurface to be considered. Then the atmospheric coupling mask should be adapted associating a non-masked index (i.e. 0) to all cells with a water fraction above the chosen threshold. The global water surface as seen by the atmosphere model is then the sum of its cell areas multiplied by its respective cell fractions. Note that invalid masked atmospheric cells should have null ocean fractions. If we follow this best practice, the atmospheric cell fractions and mask will be specific to the coupling with each particular ocean grid. A specific attribute named coherent_with_grid indicating the grid prefix of the companion grid may be defined for mask and fractions. If the OASIS API is used to define the mask and fractions, this can be done via the optional argument companion indicating the grid prefix of the companion grid (see section 2.2.4)↩︎

  16. where $(ARCHDIR) = $(OASIS_COUPLE)/INSTALL_OASIS.$(OASIS_ENV), see the header Makefiles↩︎

  17. Other measures can be obtained by defining the logical local_timers_on as .true. in different routines or by implementing other timers. Of course, OASIS3_MCT and the model code then have to be recompiled.↩︎

  18. As with OASIS3.3, the “put” calls are non-blocking but the “get” calls are blocking. As before, the user has to take care of implementing a coupling algorithm that will result in matching “put” and “get” calls to avoid deadlocks (see section 2.2.7). The lag (LAG) index works as before in OASIS3.3 (see section 2.5.2)↩︎