Up to Starting with OASIS3-MCT (first steps, tutorial, ...)
Hi All, I've done some more detailed tests of the Multi-2D case and attached some plots indicating timing costs and scalability under various different conditions. All tests with 3D fields to date simply use bilinear remapping. Basically, apart from the initialisation cost it all looks pretty good. - The "Multi2D_startup" plot (https://oasis.cerfacs.fr/wp-content/uploads/sites/114/2021/07/multi2d_startup-gif.gif) shows start-up costs for a range of different processor decompositions and numbers of fields. The blue line is for the case when I have Atoms-Ocean-Chemistry toys all configured to use 2X2 decompositions hence the label: 2x2-2x2-2x2. In this case all three components run on a single IBM node. The pink line is when I have all components configured to use 4x8 PEs i.e. one whole IBM node each - you'll see that generally the start-up cost is significantly higher in this case. The yellow line is for a case when I give the atmos and chemistry models 8x12 PEs each (so 3 whole nodes each) but leave the ocean with just 4x8 since that doesn't get involved with 3D coupling. - The "exchange" plot (https://oasis.cerfacs.fr/wp-content/uploads/sites/114/2021/07/multi2d_exchange-jpeg.jpeg) shows elapsed times for different numbers of 2-way exchanges using three different PE arrangements as above and 30 3d fields in each direction. So as more PEs are added, the actual cost per exchange comes down if we ignore start-up costs. - This is also illustrated by the "scalability" plot (https://oasis.cerfacs.fr/wp-content/uploads/sites/114/2021/07/multi2d_scalability-gif.gif) where I've plotted 1/time per exchange of 30 fields versus number of PEs used by the atmos and chemistry models. That shows a pretty good scalability line for the number of PEs I've tried. So personally, I'd be in favour of this approach compared to the full 3D field orange partition approach, for ease of set-up, use and debugging. Cheers, Richard
Hi All, Just an update on testing 3D field coupling. I've been looking at the full 3D field orange partition approach in our toys compared to the multi-2D approach. As with the multi-2D, it's scalable in terms of speeding up the exchanges when I add more PEs and the memory use scales similarly. I have run tests with the 3D case exchanging up to 99 3D fields each with 85 levels in 2-way exchanges - so effectively that's 198 3D fields in total which is equivalent to 16,830 of our "normal" 2D surface fields (our normal atmos - ocean coupling usually involves no more than about 55 2D surface fields!) - The "3DOrange_v_2DMulti_scalability" plot (https://oasis.cerfacs.fr/wp-content/uploads/sites/114/2021/07/3dorange_v_2dmulti_scalability-gif.gif) is just a comparison of the scalability of the two approaches - you'll see that they scale at the same rate but the absolute values are different (i.e. 3D is slower per exchange.) - The "3DOrange_v_2Dmulti" plot (https://oasis.cerfacs.fr/wp-content/uploads/sites/114/2021/07/3dorange_v_2dmulti-gif.gif) is just a straight comparison of absolute run times from the two approaches (running the atmos and chemistry models on 96 PEs each.) Anyway, I hope some of that might be of interest. Cheers, Richard
Hi Richard, Thanks for these results. We are working on the optimization of the coupler set up to reduce the start up costs observed with the Multi2D approach. Best regards, Laure
