REDOC_II2b

Working document on PSMILe
PRISM_init, PRISM_init_comp, wrapper codes and SCC content
S. Valcke, 01 July 2002
S. Valcke, based on Stephanie's comment, 24 July 2002

In red, text to be added.
In magenta, text to be removed.

Content

0. Introduction
1. PRISM_init and PRISM_init_comp flow charts
2. Examples
3.Deployment information needed in SCC

0. Introduction

At the PRISM First project Meeting, some users insisted on the fact that the configuration in which the different components of a PRISM coupled model can be run should be totally flexible. This means that it should be possible to run the different components

as a separate executables, or
some of them being assembled into the same executable

in the following modes:

concurrently, or
in a regular sequence (one after the other), or
in some pre-defined combination of these two modes.

Another requirement is that a stand-alone component model (one executable) should be able to run without the PRISM Driver (but it could still be required that, even in stand-alone mode, the executable is linked with MPI library). However, if a one-executable component model requires some transformations (e.g. if the forcing data on disk have a resolution different from
that of the model), the Transformer and PRISM Driver will have to be used. Furthermore, as soon as more than one component model involved, assembled or not into one executable, the PRISM Driver will be required.

For any two components being separate executables, the sequence of exchanges between them will control their execution (concurrently or sequentially).

For components assembled into the same executable, each component will be allowed to call its own initialisation and definition PSMILe instructions (PRISM_init_comp, PRISM_def_grid, PRISM_def_var, etc.), and to have its own PMIOD and SMIOC. However, there are three PSMILe routines containing global operations over the whole coupled system that must be called only once per process: PRISM_init, PRISM_setup, PRISM_terminate.

The developer will be responsible for writing the wrapping code, called the "wrapper" here after. However, the PSMILe should support all configurations. One important point here is that access to information contained in the coupling configuration file (SCC) must be possible within the wrapper after the PRISM_init call, before calling the different components.
The examples below shall be used as base for wrappers provided with the toy models (if there will ever be such toy models).

1. PRISM_init and PRISM_init_comp flow charts

PRISM_init (ierror)

if PRISM_Init has been called before, returns with a warning
if MPI_Init has not been called, calls MPI_Init
opens coupling output files (one per process implied in coupling for debugging; for production, only root process writes to stderr and stdout).
determines if the calling process is part of a standalone executable (**)
if MPI2 is used, participates to the launching the other executables

Note: After the PRISM_Init, all information contained in the SCC will be accessible, either directly (standalone case) either via the Driver (not standalone case).

(**) A particular PRISM model executable, a.out, can be started in different ways:

1.) As a single executable as one or many processes:
     > mpirun -np .. a.out
2.) As part of a coupled set of executables in a MPI1 like world
     > mpirun -np 1 driver.x -np 3 a.out -np 2 b.out -np 5 c.out
3.) In a MPI2 world as a child process spawned by the driver:
     > mpirun -np 1 driver.x (which spawns a.out, b.out, c.out ...)

In case 3.), a.out will have to be linked with the PSMILe MPI2 preprocessed (cpp) library and a fully implemented MPI2 library. For cases 1.) and 2.), a.out could be linked with either a PSMILe MPI1 preprocessed or a PSMILE MPI2 preprocessed library.

1- Distinction between case 3.) and cases 1.) or 2.):

To identify whether the application model executable is spawned or not by a parent process, the value of a intercommunicator, predefined to MPI_COMM_NULL. is checked. If the applicationmodel executable is spawned by the driver, a MPI2 call to MPI_COMM_GET_PARENT will return a intercommunicator different from MPI_COMM_NULL. Note: The call to MPI_COMM_GET_PARENT will only be included if the application model executable is linked with the MPI2 preprocessed library. In that MPI2 case, the applicationmodel executable can still be started in mode 1.) or 2.) but in thoses cases a call to MPI_COMM_GET_PARENT will return MPI_COMM_NULL since the applicationmodel executable was not spawned by a parent process.

2- Distinction between case 1.) and case 2.):

the applicationmodel executable part will perform:

standalone

logical_nodriver =.true

call MPI_Allgather(

standalone

logical_nodriver,1,MPI_LOGICAL,array_of_flags,1,MPI_LOGICAL, MPI_COMM_WORLD,error)

standalone

logical_nodriver =.not.any(.not.array_of_flags)

the driver will perform:

standalone

logical_nodriver =.false.

call MPI_Allgather(

standalone

logical_nodriver,1,MPI_LOGICAL,array_of_flags,1,MPI_LOGICAL, MPI_COMM_WORLD,error)

standalone logical_nodriver =.not.any(.not.array_of_flags)

This will detect whether there is a driver present in MPI_COMM_WORLD.: if there is a driver (case 2.), standalonelogical_nodriver will become .false. in the applicationmodel executable; if there is no driver (case 1.), standalonelogical_nodriver will remain .true. in the applicationmodel executable.

Thus the applicationmodel executable is running standalone without the PRISM Driver if the two conditions (intercommunicator == MPI_COMM_NULL .and. standalonelogical_nodriver) are met.

PRISM_init_comp (model_name, comp_id, ierror)
Stephanie would prefer the name PRISM_init_model

calls PRISM_Init if it it has not been called before
if PRISM_init_comp has been called before with argument 'model_name', returns with a warning.
If MPI1 is used, defines local communicator for the component (coloring) following the method provided by Reiner Vogelsang (SGI) implemented in OASIS 2.5.
reads all the information on the component variables in the component SMIOC; for coupling variables, receives SCC information from Driver if the calling process is part of a non standalone executable, or reads SCC information directly in SCC if the the calling process is part of a standalone executable. (Note: For a parallel standalone component, only process with rank 0 in the model local communicator should perform the SMIOC or SCC access and distributes information to the other processes.)

2. Examples

1) The subroutines for comp_x

subroutine ini_comp_x
...
call PRISM_init_comp ('comp_x', compx_id, ierror)
call PRISM_def_grid(...)
call PRISM_def_var(...)
...
end subroutine ini_comp_x
---------------------------

subroutine main_comp_x (begin_date, stop_date)
...
call PRISM_get_persist('length_of_componentrun', 'seconds', il_complength, ierror)
!
! Here the PRISM_get_persist should automatically detect the configuration in which the component is run.
! If the component is run alone in one executable or if there are more than one component in the executable
! but they are executed concurrently, it will return the length of the run.
! If there is more than one component in the executable and they are executed in sequence,
! it will return the coupling period.
!
!for models controlled by / looping over # of time steps................
call PRISM_calc_nbtstp(begin_date, stop_date, time_step_length, nb_timesteps, ierror)
do it=1, nb_timesteps
       ! comp_x physics timestep
   enddo
...
!for models being date-controlled............
actual_date = begin_date
DO
    IF ( actual_date .gt. stop_date) EXIT
    CALL comp_x_step
    actual_date = actual_date + ...
END DO
end subroutine main_comp_x
----------------------------

subroutine final_comp_x
...
end subroutine final_comp_x
----------------------------

2.A) A wrapper calling two components (comp_1 and comp_2) in sequence
Note: In this case, the length of the coupling period is fixed and the length of the run is an integer multiple of the coupling period.

program wrapper1
...
call PRISM_init(ierror)
!
call ini_comp1
call ini_comp2
!
call PRISM_setup
!
! The wrapper needs to access the length of the run and the coupling period;
! it will then be able to calculate the number of coupling timesteps it_tstep
!
call PRISM_get_persist('start_date_of_run', 'date', start_date, ierror)
call PRISM_get_persist('end_date_of_run', 'date', end_date, ierror)
call PRISM_get_persist('length_of_run', 'seconds', il_length, ierror)
call PRISM_get_persist('coupling_period', 'seconds', il_period, ierror)
      ! Returns the smallest coupling period based on coupling information in SCC.
call PRISM_calc_nbtstp(start_date, end_date, coupling_period, it_couplingstep, ierror).
     ! This routine is part of the calendar tool and calculates the number of coupling timesteps, it_couplingstep,
     ! based on the initial date, start_date,
     ! the end date of run, end_date, and the length of the coupling period, coupling_period
!
it_step=il_length/il_period
   begin_date = start_date
    do i=1, it_couplingtstep
        call PRISM_calc_newdate(begin_date, coupling_period, stop_date, ierror)
           ! This routine is part of the calendar tool and calculates stop_date by adding coupling_period to begin_date
        call main_comp_1(begin_date, stop_date)
        call main_comp_2(begin_date, stop_date)
        begin_date = stop_date
    enddo
!
call final_comp_1
call final_comp_2
!
call PRISM_terminate
!
end program wrapper1

2.B) A more general wrapper calling two components (comp_1 and comp_2) in sequence
Note: in this case, the length of the coupling period is not necessarily fixed and the length of the run is not necessarily an integer multiple of the coupling period (i.e. a partial coupling period at the end of the run is allowed).

program wrapper1
...
call PRISM_init(ierror)
!
call ini_comp1
call ini_comp2
!
call PRISM_setup
!
   call PRISM_get_persist('start_date_of_run', 'date', start_date, ierror)
   call PRISM_get_persist('end_date_of_run', 'date', end_date, ierror)
!
   call PRISM_get_persist('number_coupling_dates', 'seconds', nbr_cpldates, ierror)
      ! Returns the number of coupling dates for the run; this can be calculated based on coupling information in SCC
!
    ALLOCATE (coupling_dates(nbr_cpldates), stat=err)
    call PRISM_get_persist('coupling_dates', 'seconds', coupling_dates, ierror)
      ! coupling_dates will be a vector of integers giving all the coupling dates expressed in seconds-since-start_date_of_run
!
    begin_date=start_date
    do
        call PRISM_calc_nextcouplingdate(begin_date, coupling_dates, nbr_cpldates, stop_date, ierror)
        ! This routine is part of the calendar tool and calculates the next coupling date, stop_date,
        ! based on the actual date, begin_date, and all the coupling dates, coupling_dates.
        !
        ! if partial coupling periods are not allowed, remove comment from next line
        ! if ( end_date .lt. stop_date) EXIT(1)
        ! otherwise finish any partial coupling period:
        call main_comp_1(begin_date, stop_date)
        call main_comp_2(begin_date, stop_date)
        if ( end_date .le. stop_date) STOP
        begin_date = stop_date
   enddo
!
call final_comp_1
call final_comp_2
!
call PRISM_terminate
!
end program wrapper1

3) A wrapper calling the two components (comp_1 and comp_2) in parallel:

program wrapper2
...
call PRISM_init(ierror)
!
call PRISM_get_persist('start_date_of_run', 'date', start_date, ierror)
call PRISM_get_persist('end_date_of_run', 'date', end_date, ierror)
!
! The wrapper needs to access the number of PEs on which to run the comp_1, npe_comp1,
! and the number of PEs on which to run the comp_2, npe_comp2.
!
call PRISM_get_persist('number_process_comp1', 'nounits', npe_comp1, ierror)
npe_last_1=npe_comp1-1
npe_first_2=npe_last_1+1
npe_last_2=npe_comp2+npe_last_1
!
call MPI_Comm_rank(comm, my_rank, mpi_err)
if (my_rank .le. npe_last_1) call ini_comp1
if (my_rank .gt. ge. npe_first_2 .and. my_rank .le. npe_last_2) call ini_comp2
!
call PRISM_setup
!
if (my_rank .le. npe_last_1) call main_comp1(start_date, end_date)
if (my_rank ge. npe_first_2 .and. my_rank .le. npe_last_2) call main_comp2(start_date, end_date)
!
if (my_rank .le.npe_last_1 ) call final_comp1
if (my_rank ge. npe_first_2 .and. my_rank .le. npe_last_2) call final_comp2
!
call PRISM_terminate
!
end program wrapper2

4) A wraper to run e.g. comp_1 as a separate executable:

program wrapper3
...
call PRISM_init(ierror)
call PRISM_get_persist('start_date_of_run', 'date', start_date, ierror)
call PRISM_get_persist('end_date_of_run', 'date', end_date, ierror)
call ini_comp1
call PRISM_setup
call main_comp1(start_date, end_date)
call final_comp1
call PRISM_terminate
!
end program wrapper3

3. (Deployment) information needed in SCC (or SMIOC for standalone models)

Message passing version: MPI1 or MPI2 (???)
Number and names of executables
For each executable, the number of processes (for launching).
For each executable, the number and names of components included, and the type of execution within the executable (in sequence or in parallel; if in sequence, the coupling period is needed - coupling_period above))
For each component, the number of processes and the number of processes implied in coupling (e.g. number_process_comp1 above).
Length of run (in seconds) (length_of_run above)
Initial date of simulation
Start date of run
End date of run (static parameter)
For each coupling field, the list of coupling dates (e.g.'every 10 min starting Jan 1 00:00:00' or 'each day at 12:45:00' or 'every 10th time step starting at step 1' or 'day 15 of each month 00:00:10', and not only the length of the coupling period like in Oasis today.
...

Notes: One universal parameter is the initial date of the simulation. Another universal parameter is the initial date of the run, as one simulation may have to be split into many runs to fit the limits of he job queuing system. The PRISM System will automatically chain the different runs and will automatically change the value of the initial date of the run in the SCC or in the SMIOC. For a one-component standalone executable, there is no SCC, and all universal parameters have to be read in the SMIOC; in all other cases (including a multi-component standalone one-executable) there will be a PRISM Driver and a SCC.