Subject: Re: your mail

From: [email-masked] (Jeffery A. Kuehn)
Date: Tue, 7 May 1996 16:24:47 -0600 (MDT)

Ben--

I had only a few comments.  They are inserted into the text below.

--jeff

> 
> 
> 
> Jeff:
> 
> I've been thinking lately about how to rewrite/improve the time-gcm 
> (tgcm) model, and have written up some goals and first thoughts.
> Would you mind reading the below and send me your comments/suggestions.
> 
> I have not shown this to anyone else yet. I think that the current
> ccm2 or ccm3 is a good "model" to follow in this process, especially
> since we are currently coupling the two models together and may 
> eventually introduce tgcm mesospheric/thermospheric physics and chemistry
> directly into the ccm.
> 
> --Ben
> 
> 
> 5/7/96:
> Considerations for a rewrite of the time-gcm source code:
> 
> 
> Overall Goal:
> 
> To improve readability, modification and version control, portability,
> "plug compatability", and i/o and memory structure without sacrificing
> (and possibly improving) current levels of efficiency and performance.
> 
> All changes here proposed must have no affect on model results, i.e.,
> no changes to numerical techniques, physics, or chemistry are proposed. 
> This would be accomplished by freezing a current version of the model 
> (e.g., the upcoming new tgcm20), and make changes while maintaining 
> identical results between the new and the frozen versions. Although no
> specific changes to history file format are proposed at this time,  
> history file structure may be improved later as a result of some of
> the long term goals (e.g., portability).
> 
> Some of the below proposed changes can be accomplished soon (over
> a period of several weeks), whereas others are longer term goals
> (over a period of 1-4 years).  In general, except where noted, proposed 
> changes are given below in order of increasingly long-term goals.
> 
> Note that CRI has several performance and source code analysis tools
> to assist in this process, e.g., flint, flowtrace, perftrace, hpm, 
> profiler, etc.
> 
> 
> Proposed changes:
> 
> 
> 1. Source code and version maintenance:
> 
>    a) Elimination of the update method of modification and version 
>       control (nupdate is going away!).
> 
>    b) Maintain multi-file source with *.f and *.h or *.com files.
>       Use fortran include statement.
> 
>    c) Use either "make", or compile and load modified code ahead
>       of fixed (frozen) object.
> 
>    d) Consider use of a source code control package, although this 
>       may not be necessary given the relatively small size of our
>       research group.
> 
>    e) Build user libs for unchanging code, e.g., numerical packages.
>       (probably also not necessary initially).

It might be worth looking into moving toward Fortran 90 at this point as
well.  Getting rid of the update dependence will be very important, b & c
might almost be considered part of this.

> 
> 2. User Interface:
> 
>    a) Convert from lexrd to namelist (can be done almost immediately,
>       with 1a above).
> 
>    b) Minor changes to inputs, e.g., more flexible history mss path
>       specification and specification of external file paths for data 
>       import (e.g., boundary conditions, geophysical indices, orography, 
>       etc). 
> 
> 
> 3. Memory structure:
> 
>    a) Eliminate dependency on SSD for production runs, but retain SSD 
>       as an option (current memory resident assign of SSD units results 
>       in a total memory requirement of about 16Mw vs 6Mw with SSD)
>        
>    b) Develop an in-core version that would consume less than the 
>       current memory resident useage of 16Mw (e.g., in-core progostics,
>       out-of-core diagnostics, as in ccm). 
> 
>    c) Use of stack rather than static memory allocation.
>       (have to be careful with this one -- may have unforseen consequences)

16MW jobs are quite reasonable to run on Paiute and Aztec.  However, 16MW is
considered "large" for Shavano or Antero, and thus difficult to push through
the batch systems on those machines.  Since the availability of the SSD is
Cray specific (and in fact Y-MP specific) in our facility today, I think
eliminating the dependence is wise.

Stack allocation is another wise move, since Autotasking will require it for
best performance.

> 
> 4. I/O (see also 3a and 3b above):
> 
>    a) Consolidate use of external data files (e.g., boundaries, GPI's,
>       data imports, etc). (See also 2b above).
> 
>    b) Standardize and modularize all model/OS interface (improvements
>       have already been made in this area during the past few years).
>       
>    c) Check efficiency of pre-allocation, and compare various i/o methods.

netCDF?

> 
> 5. Code restructure: 
>    (a and b will probably require massive re-coding and are considered
>     longer term goals)
> 
>    a) Array and common block restructuring. Eliminate  the "f-array",
>       use of "pointers", and blank common. 

Again, good.  Cray Fortran 77 pointers are not compatible with Fortran 90
pointers.  It would be wise to eliminate the dependence since we hope to
move toward Fortran 90.

>    b) Improve readability and in-code documentation.
> 
>    c) In-lining of small heavily used functions, and elimination of
>       out-dated cray specific vector/CAL routines (e.g., the "Q"
>       routines, "CVMxx" functions, etc.).

yes, the compiler can do most of this now.

>    d) Above changes should allow use of compiler options such as bounds 
>       checking, and ease the use of debugger and other development tools.
> 
> 
> 6. Speed/performance:
> 
>    Goal: To accomplish the above proposed changes with no net decrease
>          in current performance levels (current code runs about 16 Mflops,
>          and is heavily vectorized).
> 

Peak speed of a Y-MP CPU is 313 MFLOPS.  I assume that the above is a typo,
since only scalar code would run as slow as 16 MFLOPS.  I suggest a great
deal of attention to efficiency at this level. The better/smoother the code
runs on one CPU the fewer problems you will encounter trying to move the
code to multiple CPUs.  Minor single CPU efficiency problems are magnified
many fold by multitasking and impossible to locate or identify when the code
is multitasked.  High single CPU efficiency is critical to successful
multitasking.  You might target something in the neighborhood of 180-220
MFLOPS single CPU efficiency before attempting to multitask.

> 
> 7. Multi-tasking:
> 
>    Goal: To accomplish the above changes as a first step toward
>          multi-tasking. I.e., multi-tasking must begin with a stable
>          fortran-standard code. Multi-tasking can be accomplished
>          at the latitude slice level (as in the ccm).

In some senses, this should almost be a secondary goal to 6, since at 16MW
the code will run single tasked on Paiute or Aztec.  Once the code is
running well on a single CPU (which will minimize you execution costs) only
then should you consider running multitasked (which is a little more
expensive, but may provide better turnaround).  To use the ND queue on
shavano, you must meet performance requirements on both 6 & 7.  It will be
worth you while to consider the requirements of 7 as you work on the others
but leave 7 till last.

> 
> 8. Portability:
> 
>    Goal: As in multi-tasking above, establish a stable standardized
>          source with as little dependence on Cray specific code as
>          possible as a first step toward running the model on workstation
>          clusters or parallel machines.

The parallel processing effort you put into the model will likely not be
very portable.  In the case of CCM2, there are actually 3 versions of it:
a parallel vector version, a data parallel version, and a message passing
version.  They are built from one source code collection, but the parallel
portions of the code are seperate modules.  The best we can do here is
probably to emphasize standard Fortran 90, and deal with the parallel issues
as they arise.

--Jeff Kuehn,  Group Head
  Technical Consulting Group
  Scientific Computing Division
  National Center for Atmospheric Research
  PO Box 3000
  Boulder, Colorado 80307-3000
  http://www.ucar.edu/consweb/TCG/kuehn/HomePage.html