
Mods to timegcm1.1 used to create timegcm1.2:

"RESULTS_CHANGE" refers to a change in the code that could affect
the model results.

!-----------------------------------------------------------------------

11/23/04: Changing all fortran stop statements to call shutdown.
Shutdown is a new subroutine in util.F that calls mpi_abort followed
by fortran stop. This was necessary for linux cluster lightning,
on which if master task did a fortran stop, the other tasks did
not know it, and the job would hang. Mpi_abort forces all tasks
to abort. From now on, to abort the program, please use 
call shutdown('message') instead of a fortran stop.

!-----------------------------------------------------------------------

RESULTS_CHANGE 11/29/04 btf: 
Fix dimensions of sco2 and sco3 dummy arguments to sub jo2h2osrb in qrj.F:

Was:
      real,intent(in),dimension(lev0:lev1,lon0:lon1) :: sco2,sco3

Changed to:
      real,intent(in),dimension(lev0:lev1,lon0-2:lon1+2) ::
     |  sco2,sco3

This was adversely affecting HOX, and later its partitioned components
OH, HO2, and H.

!-----------------------------------------------------------------------

RESULTS_CHANGE 12/03/04 btf: 
Use rmassinv_co2 instead of rmassinv_n2 in vmrco2 calculation in sub 
heatnearirco2 in heatnirco2.F:

Was:
          vmrco2(k,i) = co2(k,i)*barm(k,i)*rmassinv_n2 ! should be rmassinv_co2 ??
Changed to:
!
! 12/03/04 btf bugfix: change rmassinv_n2 to rmassinv_co2
          vmrco2(k,i) = co2(k,i)*barm(k,i)*rmassinv_co2

!-----------------------------------------------------------------------

12/03/04:
Fixed bug in dispose.F that was writing two identical msrcp commands for 
each file to dispose.csh (when namelist DISPOSE=2).

!-----------------------------------------------------------------------

RESULTS_CHANGE:
Using Hanli's bndry.F, advance.F, and addiag.F, merged with mods for
lightning. These are for kelvin and planetary waves in Z lbc. In bndry.F:
      real :: zbplanet(nlonp4,nlat)  ! planetary wave (sub bndry_planetary)
      real :: zbkelvin(nlonp4,nlat)  ! kelvin wave (sub bndry_kelvin)
This also removed a redundant addition of zb2 to Z in ncep runs (no gswm)

!-----------------------------------------------------------------------

12/01-03/04: 
Wrote new subroutine getfile (getfile.F), for finding input data files.
This allows disk paths to be given in namelist read for input data files. 
This replaces getms (getms.F is eliminated). If file is obtained from
the mss, function my_msrcp (util.F) is used. See comments in getfile.F.

!-----------------------------------------------------------------------

Changed all amin1/amax1 calls to f90 intrinsics min/max. Also changed
ifix calls to int.  This eliminates "Type conversion" warnings from some 
compilers (notably pgf90).

!-----------------------------------------------------------------------

12/06/04:
Changed output.F so that user can specify arbitrary disk path for
OUTPUT and SECOUT when DISPOSE=0, i.e., can write directly to an
arbitrary disk file when not disposing to mss.

!-----------------------------------------------------------------------

12/06/04: 
Changed sub nc_define (nchist.F) to define global attributes model_name
and model_version from tgcm_name and tgcm_version (params.F), rather 
than with values read from the source history.

!-----------------------------------------------------------------------

12/07-8/04:
Cleaned up history print to stdout in which some character variables were 
printing garbage under Linux because they were not initialized to blanks.

!-----------------------------------------------------------------------

12/22/04:
Fixed bug when reading pmlev from source history in sub nc_rdhist (nchist.F).
This bug (only when reading tgcm24 source histories) was overwriting sfeps
on lightning (was not a problem on bluesky). See comments in sub nc_rdhist
when reading variable mlev.
!-----------------------------------------------------------------------

1/4/05:
Changed longitude dimension of module data in gswm.F from nlon to
nlonp4 to avoid out-of-bounds. This was for arrays gswm_di_z through
gswm_nmsdi_vn.

1/5/05:
Added init of local elapsed_step=0. (not to be confused w/ elapsed_steps,
which is in timing.F module) in advance.F. This was prompted by lightning
(pgi 5.2), which was hanging on a print to stdout of elapsed_step. After
explicit init to zero, lightning did not hang on the print statement. 

!-----------------------------------------------------------------------

1/05: Rewrote timing.F -- this changes all timing/timer calls 

!-----------------------------------------------------------------------

1/05: Changed array() to wkarray() and made wkarray module data in
dynamo. This was to avoid out-of-bounds in array().

!-----------------------------------------------------------------------

RESULTS_CHANGE 1/26/05:
Fixed bug in dynamics, in which difkk was passed to comp_major as
(levd0,lond0,lat) instead of (levd0,lond0,latd0).

!-----------------------------------------------------------------------

Continued improvement of in-house timing, see timing.F, advance.F,
and dynamics.F. Also added several fields to secondary histories
when "diags" is set true in dynamics.

!-----------------------------------------------------------------------

Force shutdown if NaN's are found when writing histories, see nchist.F:

      nanfatal = 1 ! abort if nans are found
      if (check_nan) 
     |  call check_nans(f3d,nlon,nlat,nlevp1,name(1:8),nnans,
     |    0,0.,1,nanfatal)

Also see sub check_nans in util.F.
This is to avoid runaway jobs that are writing NaNs to histories.

This is set on bluesky only -- I do not know how to detect NaNs from
fortran under pgf90 yet, so runaway jobs are still possible on lightning.

!======================================================================

Jan 31, 2005: 
Committing to CVS and tagging as timegcm1-1-1. This includes the above 
mods to timegcm1.1, but is before adding Ray's tuning mods to timegcm1.1. 
After this tag, Ray's tuning mods to timegcm1.1 (yr92ncep) will be added, 
a new timegcm1-2 tag will be created, and version timegcm1.2 will be released.

>>>>> MODS BELOW THIS LINE ARE APPLIED TO timegcm1.1.1, to make timegcm1.2 <<<<<

!-----------------------------------------------------------------------

2/1/05:
Merging Ray's tuning mods to timegcm1.1 (yr92ncep/yrrun) with timegcm1.1.1, 
to make timegcm1.2. (Ray's mods originally copied from 
  ~roble/timegcm/timegcm1.1/yr92ncep/modsrc.yrrun)

Merged changes that do NOT affect results:
(however, changes in bndry.F may affect results in non-standard modes,
e.g. running without gswm, turning on planetary or kelvin waves, etc):
  addiag.F, bndry.F, and duv.F

MAJOR RESULTS_CHANGE 2/1/05 (gravity wave tuning):
Merged changes that DO significantly affect results:
  gw_drag.F, gw_share.F, mgw.F

!-----------------------------------------------------------------------

2/3/05: Fixed problem where certain scalars were not being written to
the histories properly. Also removed call hist_initype in output.F,
because this call was wiping out values read from the source history.
!-----------------------------------------------------------------------

2/3/05: Ran serially on hao linux sundog, and fixed some timing calls
with MPI not defined. 
!-----------------------------------------------------------------------

RESULTS_CHANGE 2/3/05:
Added interpolation of zatmos t,u,z boundaries to current model time.
Previously, these were updated only at day boundaries. See new zatmos.F.
Currently, only zatmos_tn is used (see dt.F). Changes to results will be
fairly minor, especially for short runs of a few days or less.
!-----------------------------------------------------------------------

2/7/05: Removed sum1a and sum1exp diagnostics from qrj.
!-----------------------------------------------------------------------

2/8/05: Committing to CVS and tagging as timegcm1-2. This will be saved
in $TGCMROOT at hao and scd as timegcm1.2.

!======================================================================
Continued mods after timegcm1.2:

3/7/05: Adding user-provided time series for f107 and f107a, analagous
to ctpoten and power time-series. Implemented all 4 time-series options
in advance (this had apparently not been implemented in timegcm1.x).
See input.F and advance.F (e.g., f107_time, ntimes_f107, etc).

4/14/05: Changes to radcool.F to save cooling term diagnostics in
deg/day (as per roble request).

4/21/05: Added new solar proton event for Jan 15-22, 2005 to protons.F, 
as per Jackman data (see ~/usr/jackman). Commented out the check for year 
in protons.F, so it can be run with ncep from a previous year.

Noticed that fields.F had been changed April 6th, inadvertently uncommenting
the allocdata call. This meant that  allocdata was called twice, once from
fields.F and once from init, resulting in allocation errors. Not sure how
this happened, but removed the call from fields.F.

Sometime in May: added meped option to read MEPED data.

7/8/05: Changed length of line read by sub rmcomments (util.F) from 
80 to 6400, so namelist read lines in .inp file can be long (thanks to 
Cedric St-Jean).

7/27/05: Committing to repository, and tagging as timegcm1-2dev1

7/27/05: Merged Astrid's lamdas/qjoule mods. The following files
were new: dynamics.F, fields.F, lamdas.F, qjoule.F. Saved this as
CVS tag timegcm1-2dev2 (also available in $TGCMROOT/dev/timegcm1-2dev2).

8/18/05: Updated util.F with new sub expand_path, and added call to
expand_path in getfile.F. Also updated getfile.F from tiegcm.

12/5/05: Opened up year check statement in protons.F (see 4/21/05 comment above)

12/23/05: Added call filter() option to oplus (see callfilt1 and callfilt2
in oplus.F).

WARNING: will not compile right now with hres=2.5 and vres=0.25, i.e., double
res, see orogdat.F.

!-----------------------------------------------------------------------
4/19/06:
Wrote /home/tgcm/zatmos/mkdres.pro to interpolate zatmos data to 2.5 deg
for double res timegcm.
!-----------------------------------------------------------------------
5/3/06: 
Starting work on update to timegcm1.3. Most of this work will follow 
tiegcm-hist-lbc.

Updated scripts directory as in tiegcm-hist-lbc.

Changing gswm as in tiegcm-hist-lbc (gswm.F, init.F, input.F, bndry.F, etc).

Major changes to many files, esp hist.F, nchist.F, output.F for new format
histories.

5/4/06: 
Completed first stage of new format histories for timegcm1.3. Saved tar
file /e/foster/timegcm/timegcm1.2.1.tar. This code makes new format
histories, but crashes w/ NaNs, probably because of lbc in top slot.
This does not yet have addflds for secondary histories.

5/15/06:
Fixed "top slot" in minor2.F. This fixed above problem coming out of comp_hox.
Several other changes, including gpi_ncfile, and minor changes to nchist.F,
and mgw.F, gw_drag.F. Still does not have addflds for secondary histories.

5/16/06:
This code is now running, altho there are differences with raw timegcm1.2.
Some diffs may be due to above mods from 3/7/05 to 12/23/05, but not sure.
Diffs may also be due to tgcmproc_f90, midpoints vs interfaces, etc.
Some diffs (O, hydrogen and nitrogren sp) have diffs at lbc, but they 
actually look better in the new code.

Since full fields from the new code look basically ok, even better than
timegcm1.2 at lbc of composition sp, I am going to save this before moving 
on to secondary history changes, i.e., addflds.  Still have not merged Ray's 
tuning mods (modsrc.ncep).

5/16/06: Committing to repository, and tagging as timegcm1-2dev3.
(Previous tag was timegcm1-2dev2, committed 7/27/05, see above).

5/18/06: 
Have added addfld and changed nchist, mpi, output, etc accordingly.
Also wrote perl script (src/addfld/addfld) to convert addfsech calls to
addfld calls. Did this in qrj only, but it seems to work with a little
cleanup after addfld processing. The addfld perl script may not work for
certain addfsech calls, e.g., in smooth.F:

!       if (isave > 0)
!    |    call addfsech('SMOO1',' ',' ',ftmp(:,i0:i1),
!    |      i0,i1,k1-k0+1,k1-k0,j)

Copied addfld to scripts directory.

Removed sub qrj_exp from qrj.F, and introduced check_exp for a few
expo calls. A few other calls use expo even when check_exp is not set.

Committing, but not tagging at this time.

5/19/06: Translated addfsech calls to addfld calls in mgw.F, dt.F,
and duv.F.

5/22/06: Added subs qrj_diags_inp, qrj_diags_pd, and qrj_diags_q as
internal to sub qrj. These routines call addfld to save groups of
fields to secondary histories.

5/24/06: Finalizing per scripts addfld and addfsech2addfld. The former
operates on addfld calls in input files, the latter converts addfsech
calls to addfld calls. These are in the scripts dir. (also in ~/tgcm/tiegcm-hist-lbc/
scripts.)

5/25/06: Found some incorrect comments re midpoints/interfaces in
comp_ox.F and comp_hox.F. Trying to be consistent with var names
ending in "i" to indicate vars converted from midpoints to interfaces,
and var names ending in "m" for vars converted from interfaces to
midpoints. This is generally important for vars that are use-associated
from qrj, where they are calculated at interfaces and interpolated to 
midpoints for use in other routines such as comp_ox.F and comp_hox.F.

Am working through all composition routines, converting addfsech calls
to addfld calls, saving and looking at secondary history fields for
each routine. Using perl scripts addfld and addfsech2addfld.

5/30/06:
Note there are 2 ways to save secondary history fields only up to
level lev1-1 (these examples are from comp_ox.F):

1. Pad the top level (lev1) with special value ("missing data"):
      use params_module,only: spval
      ...
      fs11(lev1,:) = spval
      call addfld('FS11' ,' ',' ',fs11,'lev',lev0,lev1,'lon',i0,i1,lat)

2. Tell addfld you want to save only up to level lev1-1 (addfld initializes
   whole array to spval):
      call addfld('O3_O'    ,' ',' ',o3_o(lev0:lev1-1,:),
     |  'lev',lev0,lev1-1,'lon',i0,i1,lat)

In both examples, the top levels will contain spval on the netcdf history
file. Of course, you can save the whole array, with garbage in the top 
level, and simply not plot that top level, but one of the above methods
is better. The significat difference in the 2 examples is that the first
changes the local array, and the 2nd one does not.

5/31/06: Completed addfld calls in comp*F. Am committing, but not tagging.

Translated addfsech to addfld in dynamics.F (diags==.true.)

6/1/06: Completed addfld calls in dynamo.F, lamdas.F, advance.F.

6/6/06: Completed addfld calls in chapman.F, advec.F, cpktkm.F, current.F, 
divrg.F, efield.F. 

6/21/06: Have finally converted all addfsech calls to addfld.
         Am committing and tagging as timegcm1-2dev4.

!-----------------------------------------------------------------------

6/22/06: Merging Ray's mods (~roble/timegcm/timegcm1.2.ncep/modsrc.ncep)
 - aurora.F: mostly mods for protons, plus minor tuning
 - protons.F: minor adjustments as per aurora
 - chemrates.F: tuning mods
 - comp_meta.F: add e5200, e6300s, ro1dn2d, xo1d, xn2d
 - dt.F: add thermo_diags
 - gw_drag.F: tuning mods 
 - hoxpion.F: comment out conditional return if solar_protons <= 0
 - lamdas.F: Astrid's mods re omegas, geoNao. Also lamda1 is now output
 - oplus.F: change parameters phin, ppolar to -2e8 (retain callfilt1,2)
 - qjchem.F: add chem heating diagnostics
 - qjoule.F: mostly comments re Astrid mods
 - radcool.F: small change in cool_no calculation
 - solgar.F: add multipliers 
     (was for co2 double study, but currently set to 1., so no effect)

Testing this code by repeating the last 5 days of Ray's 2005 timegcm1.2 
run (days 95-100) (with ncep and gpi).

6/23/06: Comparison with the timegcm1.2 py5n case looks good, so am
committing this and saving cvs tag timegcm1-2dev5. Will give this to
Ray for further testing. (Makefile is now building with -q64 for
64-bit addressing).

!-----------------------------------------------------------------------

7/7/06: Merged ~roble/timegcm/timegcm1.2.ncep/modsrc.ncep to work for
timegcm1.3 (i.e., ~foster/tgcm/timegcm/src), but leaving "tuning mod"
files, see ~foster/tgcm/timegcm/src/modsrc.new. Also see filemerge.csh
in ~/tgcm/timegcm/src.

10/9/06: Merged ~roble/timegcm/timegcm1.2.heat/modsrc.ncep to current
source. Only changes were in mgw.F (Hanli's units fix that raised TN
and inspired Ray's new runs of 2002 and 2004 (so far -- he also has
plans to make new runs for 2003 and 2005 as well). The other change 
was adding new solar protons event to protons.F.

10/12/06: Wrote double resolution solgar_bndry_2.5deg and solgar_import_2.5deg
(the import file is also double res in vertical, i.e., dlev = 0.25).
These are now being read by this code in double-res timegcm. However,
the double-res code is seg-faulting on step 13.

10/12-13/06: Set kut in cons.F according to NLAT (defs.h). This removes
the need for sub set_wave_filter. This is as in tiegcm-hist-lbc.
Also changed 0's at extrems of kut in cons.F, to 1's.
Also set dipmin (cons.F) and sin10 (magfield.F) according to NLAT,NLON.

10/16/06:
Now using NLAT,NLON to set the "nn" array in sub filter2 (filter.F).

Introduced Liying's new qrj.F. This appears to be working at least
in the single-res configuration. I doubt it is related to the NaN
problems at double-res. Using the new qrj requires minor mods in
tgcm.F (no calls to init_euvac or init_sigmas from tgcm.F), and
dynamics.F (no call to xray). Also converted addfsech calls in Liying's
version to addfld calls.

10/17/06: Set logicals at top of oplus module true, so oplus is calling
both filter and filter2.

At double-res, this code is still producing NaNs in TE and POTEN after 
about 5-6 minutes at a 30-sec timestep. Will commit and tag this, and 
give it to Ray for tuning. The tag will be timegcm1-2dev6.

Re-committing and retagging after adding the following files to cvs
repository in scripts directory: Mknamelist, Mkdepends, Mklogs, Mklist,
Mkdispose. Also cvs added addfld.F to src dir. Will overwrite timegcm1-2dev6 tag.

!-----------------------------------------------------------------------
10/18/06:
Added sub filter_uvbnd, called at end of sub tuvbnd, tuvbnd.F.

10/24/06:
Fixed bottom boundary of tni in duv.F:
> !         tni(lev0,i) = tn(lev1,i,lat)  ! lbc of tn is stored in top slot
>           tni(lev0,i) = tlbc(i,lat)     ! lbc of tn is in tlbc

Also in qrj:
> !       tni(1,i) = tn(lev1,i) ! tn bottom boundary is stored in top slot
>         tni(lev0,i) = tlbc(i,lat)

Also rewrote $TGCMDATA/TGCM.timegcmdr.peqmb015.nc with /home/tgcm/mksrc,
with a fix for TN_NM, UN_NM, VN_NM (see /home/tgcm/mksrc/README). This
fix retains lbc in the top slot for these 3 fields (also TN,UN,VN), 
thus keeping both the sres input and dres output files in "old" history 
format, so the model/proc will find lbc in the top slot (mksrc does not 
add TLBC, etc, i.e., its output files are still "old" format).

Will commit this and update Ray's timegcm1.3 directory. 

!-----------------------------------------------------------------------
1/11/07:
Add option to read ncep reanalysis data. See /home/tgcm/ncep_reanalysis.

1/18/07:
Astrid noticed that running tiegcm1.8 with MPI, but only a single task,
that the histories were not changing. This was because foutput was not
getting updated. Fixed this by adding elseif to the following conditional
in advance:

#ifdef MPI
      if (time2write.and.ntask > 1) then
        call mp_gather2root(itc,'prim')
        if (wrsech) call mp_gather2root(itc,'sech')
      elseif (time2write.and.ntask==1) then
        do i=1,nf4d
          foutput(:,lon0:lon1,lat0:lat1,i) =
     |      f4d(i)%data(:,lon0:lon1,lat0:lat1,itc)
        enddo
      endif
#else

1/19/07: ncep_reanalysis now working. This includes changes to advance.F,
input.F, addiag.F, dt.F, and new file ncep_reanalysis.F.

Code has been tested from day 80 with both ncep_reanalysis and gpi.F,
and Ray's mods. Seems to work ok, altho sometimes STEP has to be reduced
from 120 to 90, or even 60.

1/22/07: 
Adding Ray's tuning mods in bndry.F, filter.F, gw_drag.F, gw_share.F,
mgw.F, and qrj.F from 1/19/07. See save/*.pretune for these files before 
Ray's tuning mods.

Commit and tag as timegcm1-2dev7. Astrid and Maura will use this code
while Ray continues to tune.
!-----------------------------------------------------------------------

1/25/07: Add ecmwf_ncfile -- data file from ECMWF ERA-40 with T,Z
lower boundaries. Now there are 4 options for T,Z boundaries:
  1. ncep_ncfile (use ncep data from Bill Randell, see /home/tgcm/mkncep)
  2. ncep_reanalysis (use ncep reanalysis data, see /home/tgcm/ncep_reanalysis) 
  3. ecmwf_ncfile (use ECMWF ERA-40 data, see /home/tgcm/ecmwf_era40)
  4. none of the above (hough modes from bndry.F)

1/26: added ncep_reanalysis and ecmwf_ncfile file names to timegcm1.3 histories.
Both files now have Z in cm (the conversion, which is different for ecmwf vs
ncep files, is now done by the ncl scripts used to write the files from the
original datasets, see /home/tgcm/ncep_reanalysis and /home/tgcm/ecmwf_era40).

about 2/1/07: Now using ECMWF TOGA data (ds111.2). See /home/tgcm/ecmwf_toga.
Either ecmwf_era40 or ecmwf_toga files can be provided for ecmwf_ncfile. 

2/5/07: Adding U,V to ecmwf_toga file (ecmwf_ncfile) (ecmwf_era40 file still
has T,Z only). Now adding mods to use U,V from ecmwf_toga instead of tuvbnd.F.

2/6/07: Mods to ecmwf.F, advance.F, dynamics.F now use T,U,V,Z from ecmwf for
lbc of these fields, if ECMWF_NCFILE = $TGCMDATA/ecmwf_toga_2002060-2002090.nc.
In this case, tuvbnd is not called.  See /home/tgcm/ecmwf_toga for how to construct 
the ECMWF TOGA data file.

2/9/07:
If calendar_advance==0 and ECMWF_NCFILE is set, then call ecmwf_bndry
with start_day and ut=0 (see call in advance.F). Otherwise, if calendar_advance > 0
then call ecmwf_bndry with current model day and time.

2/22/07:
Changed hard-wired "36" to "nlat/2" in sub gwsource (mgw.F). This makes
it generic for different latitudinal resolutions (the 36 would only work
at 2.5 res, where nlat==72)

Added longitude dimension to raykb and raykccm arrays in sub gwsource to
avoid discontinuities at subdomain longitude boundaries caused by setting
"arbitrary longitude" lonz = lon0. The addition of longitude loops prevents
this discontinuity. This was first noticed in ce projection plots of RAYK
(rayfrc, derived from raykk and raykb). See mgw.iloopfix.

A better solution may be to use zp instead of z in the raykccm calculation.
This eliminates use of Z, and does not require addition of longitude dim
as in the solution above. It does remove the latitudinal variability that
was coming in from Z, however if a lat function is needed, it can be added
using ablat (which is currently commented out). See mgw.zpfix.

Added i0:i1 to addfld call for GWUU in duv.F. This prevents 0 values
between mpi task longitude boundaries appearing in the GWUU diagnostic.

Feb-May, 2007: Have been developing parallel dynamo mods in modsrc.pdyn.

5/24/07:
Added use of u,v lbc from ncep_reanalysis data file. The modified files
were advance.F, dynamics.F, and ncep_reanalysis.F. See conditional around
the tuvbnd call in dynamics.F. Also note that ncep_reanalysis reads units
attributes from the netcdf data file, and makes unit conversions on Z
(m to cm), U (m/s to cm/s), and V (m/s to cm/s), if necessary. Ray will
use these mods in his timegcm1.3.ecmwf/modsrc.ecmwf dir for Jan 1992 run
for Gordon Shepard.  See also /oreo/d/foster/timegcm/modsrc_ncep_rean.  

Also see /home/tgcm/ncep_reanalysis for the procedure to write the ncep_reanalysis
data files. See regrid.ncl for whether or not Z,U,V units are changed there
or not, but as mentioned above, the model checks the units attributes on the
files and acts accordingly.

7/20/07:
Add step_qrj (see input and qrj call in dynamics)

1/4/08:
Mods to several files to insure that quantities calculated by qrj.F (total heating,
certain ionization rates, etc) are incremented by various heating terms in other
routines called after qrj only if qrj has been called for the current step. Also 
changed the step_qrj conditional to insure that qrj is always called in the first 
step. Files changed are: advance.F, aurora.F, dynamics.F, heatnirco2.F, qinite.F, 
qjchem.F, qjion.F, qrj.F, qtieff.F, settei.F, xray.F. Grep on step_qrj to see most 
of these changes.  See also ~foster/timegcm/timegcm1.3-qrjstep/modsrc.eqdms.

1/29/08:
Added writes of calendar_adv and write_date in nchist.F. These variables
were defined on the history, but data not actually written.

5/27/08:
Some of Ray's timegcm1-2dev7 jobs were hanging when certain combinations
of prognostic and addfld secondary history files were requested. Tracked
this down to sub mp_gather2root_sechist in mpi.F. This sub was improperly
referencing ilon0,ilon1 and ilat0,ilat1 in the send and receive buffers
(sndbuf and rcvbuf). Fixed this to use nlons and nlats, similiar to
sub mp_gather2root_prim. This bug was introduced in November, 2005,
when mp_gather2root_sechist was rewritten for the addfld method.

10/15/08 btf:
Removed the following from qrj.F:
      real :: rlmsrcinv(l1-1)
      ...
      do i=1,l1 
        rlmsrcinv(i) = 1./rlmsrc(i)
      enddo
This was going out of bounds in rlmsrcinv when i==l1. Since rlmsrcinv
was not used, I removed both the declaration and the above loop.

10/20/08 btf:
Zero out upper boundary condition xxx_ubc in comp_ch4, comp_co, and
comp_co2. It is zero in all species except hox, but was not zeroed
out in ch4,co,co2. This is important because sub minor does use these
zero values, and since xxx_ubc is allocated dynamically, it would have
garbage if not set. Apparently, the values happened to be very small,
e.g., 1e-300, so it was not noticed, but when running double-res on
bluefire with 64 pes and ptile=64, some real garbage got in there
and screwed everything up. The symptom was that it was crashing in
fft9.F.

Also changed "1" dimensions to "*" in mudmod.F (PE(NXX,*)) and dynamo.F
(sub ceee), to allow the runtime out of bounds checker -C to be turned
on without stopping at these places.

10/21/08 btf: Should set p0_model if it was not read from source history.
Currently, p0_model is 1.e36 if not read.

11/13/08 btf: Units and vcoord were wrong for O+ (OP) -- set units to CM^3
and vcoord to midpoints in fields.F. Vcoord was also wrong for OP_NM.

Unknown date: added step_compqrj as a "split" timestep option.

------------------------------------------------------------------------
1/15/09:
Merged modsrc.inits to working dir. These are init of dynamically allocated
arrays (e.g. comp*F), new dyanmo w/ Astrid je1,je2 currents, declare
zpotenm3d as pointer rather than statically in dynamo module. Zpotenm3d
is allocated in allocdata.F.

defs.h correction to ZMBOT,ZMTOP at double res. Note new upper boundary
to zp +7 not yet added.

Add magpres_g.F as new source file (did cvs add).

Did cvs commit -F commit.notes/commit.notes.jan15-09 (not tagging).

1/15-16/09:
Adding mods for new ubc zp +7 (up from +:5), see defs.h, aurora.F, params.F,
tgcm_config, Mknamelist (aurora.F was copied from modsrc1.3n.ecmwf). 
With zp +7 ubc, NLEV==48 or 96.

Made new source files (with zp +7 ubc), put in $TGCMDATA at CISL, and 
fixed Mknamelist for all data files for both resolutions.

Opened addfld calls for UI_VEL,VI_VEL,WI_VEL in ionvel.F. These are
included in SECFLDS in Mknamelist (default namelist input file).

Tested on bluefire with short runs at both 5.0 and 2.5 deg.

1/17-20/09: Merging modsrc.ncepb02
Later: Also merging parts of modsrc1.3n.ecmwf

2/4/09: 
Copied *.F from working dir on bluefire:/ptmp/foster/timegcm/new1/timegcm/src,
and scripts from the same directory.

This code is the result of extensive debugging related to dynamically allocated 
memory. The code was chronically seg faulting, but it would come and go at unexpected 
times (e.g., adding print statements near the problem might cause it to succeed). 

Shortly before this code started working, NaNs were appearing in U,V in the first 
timestep, and the code was seg faulting in the third timestep.  Many allocatable 
arrays and pointers were given the "save" attribute, even when declared as module 
data. See several source files, but especially fields.F. This code is also a result 
of merging code from modsrc.ncepb02 and modsrc1.3n.ecmwf, altho this is probably not 
complete yet. In any case, this code appeared to be working on bluefire at both single 
and double resolutions, and using 64 pes with or without SMT=64. See also email 
correspondence with Sidd Ghosh during January-February 2009.

State of src before the copy from bluefire was saved as src_feb4-09/ and 
src_feb4-09.tar, and scripts was saved as scripts_feb4-09/ and scripts_feb4-09.tar. 

Double-resolution run w/ STEP=90 seg faulted after about 20 hours, but it
succeeded when I ran it w/ STEP=30, and turned on GPI and ECMWF after realizing that
the source history was from a GPI and ECMWF run. 

2/5/09: Added mkhvols.F module from tiegcm, and removed function mkhvols from
input.F (which was not working). Calls to mkhvols in input.F are same as before.

Added initialization of f4d%data and f3d%data to fields_initval=0., see fields.F

2/13-14/09: Code was hanging in one of the mp_bndlons, after changing to one-time
buffer allocations in several mpi.F subroutines.  Problem was related to
different widths of longitude dim subdomains when running w/ 64 procs (#lons
varies between 18 and 19), so the buffer sizes were different for some tasks
after the first call when they were allocated. This was easily fixed by using
mxlons in the buffer allocations. Also several changes to calls and dummy arg
dimensioning in several mpi.F routines. Using Sidd's Makefile.malloc (see scripts/)
which inits dynamically allocated memory to NaNs. This, in combination with
the usual debug flags, helps debugging.

------------------------------------------------------------------------

3/6/09: 
After considerable more effort improving dynamic memory security at 5.0 
and 2.5 deg resolutions, 64 pes with and without SMT, am ready to commit 
and tag as timegcm1-4. 

This is not a finished code, rather its a working waypoint. New subversions
1.41, 1.42... should follow soon with improvements to efficient use of MPI,
and other performance improvements; also option to write 32-bit secondary
histories, and eventually netcdf4.0.

The cvs tag timegcm1-4 represents primarilly the following changes:

-- Merge Roble's source modification directories (modifications to timegcm1-2dev7): 
     - modsrc1.3n.ecmwf (used with 2.5 deg res), 
       (See $TGCMROOT/timegcm1-2dev7/src/modsrc1.3n.ecmwf)
     - modsrc.ncepb02   (used with 5.0 deg res), 
       (See $TGCMROOT/timegcm1-2dev7/src/modsrc.ncepb02)
-- Add save attribute to allocatable arrays, and add initialization after
   allocation (see esp fields.F, mpi.F).
-- One-time allocation of receive and send buffers in mp routines (see esp mpi.F)
-- Extension of upper boundary from zp +5 to +7.
-- Add ECMWF low boundary data option (6-hrly data, 2.5 deg res only)
-- Save debug AIX Makefiles in scripts directory (Sidd Ghosh malloc lib)
-- Code tested on AIX bluefire at 5.0 and 2.5 deg resolutions, with up
   to 64 pes, with and without SMT.
-- Code builds and runs under HAO Linux oreo system: 
   Linux oreo 2.6.23.17-88.fc7 #1 SMP Thu May 15 00:35:10 EDT 2008 i686 i686 i386 GNU/Linux

State of working dir /home/foster/tgcm/timegcm, with modsrc and development
dirs,  was saved in /home/foster/tgcm/timegcm_pre1.4.tar.gz before doing
the cvs commit and tag timegcm1-4. Modsrc directories saved in this tarball
include the following (modsrc1.4 being the "final" product of the merges):

drwxrwxr-x 2 foster tgcm   4096 Jan 22 13:10 modsrc.inits/
drwxr-xr-x 2 foster tgcm   4096 Jan 20 12:56 modsrc.ncepb02/
drwxrwxr-x 2 foster tgcm   4096 Oct 13 15:23 modsrc.pdyn/
drwxrwxr-x 2 foster tgcm   4096 Jan 23 10:56 modsrc.ubc7/
drwxr-xr-x 2 foster tgcm   4096 Feb 10 15:56 modsrc1.3n.ecmwf/
drwxr-xr-x 2 foster tgcm   4096 Mar  6 11:50 modsrc1.4/

3/6/09: 
Executing cvs commit -F commit.notes/timegcm1-4, followed by 
cvs tag timegcm1-4.

8/3/09:
Turns out that svn tag timegcm1.41 is unstable, but timegcm1.4 is stable.
So this is a new svn branch, starting from 1.4 code, to incrementally
reintroduce 1.41 changes, testing along the way, with the goal of retagging
a stable timegcm1.41. The idea is to start w/ non-lbc changes (e.g.,
consolidation of MPI calls), and make sure that is working before
reintroducing the new lbc code. 

8/3/09:
Consolidated mp_periodic_f3d calls, tested on bluefire, and committed rev118 

8/4/09:
Consolidated mp_periodic_f2d calls, tested on bluefire, and committed rev119

Consolidated mp_bndlats_f2d, mp_bndlons_f2d (duv.F), and mp_bndlons_f3d 
calls (several). Also added mpi_timing (see routines at end of mpi.F, and
report call in tgcm.F).

Reinstated mpi_timing (mpi.F)
Changed nf_get_var calls in gswm for ifort compiler

~7 pm: Svn commit of rev120

8/5/09:
Additional changes for ifort compiler:
 - Remove ilink function (util.F, dispose.F, putms.F, getfile.F, output.F)
 - Fix nlatrd check in solgar.F
 - Add serial periodic points in comp_noz.F

Additional changes for mpi_timing (advance.F, timing.F)

Code tested on the following systems:
 - 32-bit Linux "oreo", serial with pgf90.
 - 64-bit Linux "cerberus", MPI 4-procs with pgf90.
 - DASG Linux "gale", serial with ifort.
 - AIX "bluefire", MPI with xlf90 (5-day 64-proc runs at sres and dres)

svn commit around 2:30pm 8/5/09.

Tested default run on Linux MPI lightning with pgf90
(used $TGCMROOT/bin/tgcmtest to do a svn checkout and
submit the job).

8/6/09:
Enforce noz_out >= 1.e-10 in comp_noz.F. 

! Comments in sub minor, comp_noz.F:
! 8/6/09: noz_out can go to zero. This does not cause the model
!         to crash, but in some cases can cause tgcmproc_f90 to
!         hang in cplbam (sub contour in plt.F) when contouring 
!         log10 of NOZ, but sometimes it works, properly inserting 
!         missing values and plotting white boxes in the plots. 
!         Saw this at equatorial latitudes around zp -12 in slt 
!         noon longitude slice.

Tested on bluefire with 5-day runs with NCEP (5 deg) and ECMWF (2.5 deg).

Removed timegcm-sgi.job from scripts directory.

This code still does not have the new lbc code (lbc.F), so its looking
like the problem w/ the original 1-41 tag was related to the new lbc code.

Committing to svn (will be rev 122).

8/7/09:
Have introduced lbc.F and gradually moving from old to new lbc.
Both old and new lbc are being called in the same code, and are
being compared in addiag.F (for Z), duv.F (for U,V), and dt.F (for T).
See temporary source file compare.F, takes 2 2d fields, counts and
saves diffs. Am seeing diffs tnlbc-t_lbc around +/-1.e-13, and
diffs z(1,..)-z_lbc around +/-0.5e-9. Diffs of u,v are zeroes,
because they both come from the same tuvbnd output.

Am now using t,u,v,z_lbc in addiag, duv, dt. So far have tested
zatmos+gswm at single and double resolution. When starting to use
the lbc fields, the differences w/ tnlbc and znlbc appear along
distinct latitudes corresponding to processor boundaries. This
may be related to exchange of halo lats, i.e., mp_bndlats calls.

Added sub inp_lbc to input.F to enforce lbc rules. Can now begin
testing lbc options NCEP, ECMWF, etc.

Added test in sub inp_lbc (input.F) to enforce one and only one 
lbc data file can be specified: zatmos_ncfile, ncep_ncfile,
ecmwf_ncfile, or ncep_reanalysis. (e.g., cannot use zatmos with
any of the others).

8/7/09 (3pm):
Code is now running both old "bndry" and new "lbc" boundary methods.
Am comparing old and new, and can control which is actually used.
See calls to sub compare (new temporary source file compare.F) in
addiag, dt, and duv. Right after these calls is code to use the
new lbc, i.e., t,u,v,z_lbc. This appears to be working for zatmos+annual 
tide and ncep and ecmwf options. Saved src/bndry_and_lbc.tar.gz.
Now will start cleaning up, i.e., removing debug print statements,
gradually removing the old lbc code, and using t,u,v,z_lbc throughout,
including diags, and hist vars TLBC,ULBC,VLBC.

8/10/09:
Added temporary flag lbc_code (lbc.F) to indicate whether to use
the old lbc vars (z(1,..),tnlbc,unlbc,vnlbc), or the new lbc vars
(t,u,v,z_lbc from lbc.F). This code still calls compare.F to get
diffs between old and new vars.  Ran the following tests:

sres zatmos new and old code (5 days)
sres ncep new and old code (5 days)
dres zatmos new and old code (1 day)
dres ecmwf new and old code (1 day)

tgcmproc_f90 differences from output histories are essentially zero, 
so am saving this (will be rev 123).

See bf:/ptmp/foster/timegcm1.41_branch with subdirs for dres and proc.
See also /oreo/d/foster/timegcm1.41_branch.
Working svn dir is ~/tgcm/timegcm/branches/timegcm1.41.

8/10/09: Committed rev123, but had to add lbc.F, so recommitted,
saving rev124.

8/11/09: Carefully backing out old boundary code bndry.F, conditionals
in addiag, dt, duv, etc., while using new lbc.F code.
Remove: bndry.F

8/13/09: Have removed old boundary code, now using only new lbc code.
Remove bndry.F and tuvbnd.F, and add uvbnd.F.

Committed svn rev 125. Astrid is now using this for testing lbc.

8/17/09: Checked out svn r125 of branch timegcm1.41 to $TGCMROOT
(hao and bluefire) for Ray to test. The SVN command was:
"svn checkout $SVN/timegcm/branches/timegcm1.41 timegcm1.41"

8/17/09:
Added namelist input parameter sech_nbyte. This must be 4 or 8.
SECH_NBYTE=4 means write 4-byte reals to secondary histories, 
SECH_NBYTE=8 means write 8-byte reals to secondary histories.
Default is 4. Changes to input.F and nchist.F. This was tested
on bluefire, and tgcmproc_f90 showed 0 diffs between 4-byte and
8-byte fields. This setting applies only to the full 4-d fields.
The 4-byte secondary files are almost 1/2 the size of the 8-byte 
files. Primary files are unaffected by sech_nbyte.

Note only sub defvar_f4d in nchist.F, and calls to it, was changed,
because it is only necessary to change the definition of the variable.
The put_var_double calls are unchanged, i.e., we are still passing
doubles, but telling netcdf to convert the 8-byte doubles to 4-byte
reals before writing them to the file.

8/17/09: Tagged timegcm1.41, merged branches/timegcm1.41 back to the
trunk, and removed the branch. Also copied nchist.F and input.F to
the working copy of the trunk so it has sech_nbyte.

8/18, 8/19:
Merged Ray's timegcm1.3.eqdgc/modsrc.eqdnc mods to work with 1.41.
See ~/timegcm/timegcm1.41/modsrc.eqdnc. Also note Ray's entire 
timegcm1.3.eqdgc directory was copied to ~/timegcm/timegcm1.3.eqdgc, 
for reference.

8/19/09: Added additional checks on sech_nbyte in nchist.F for
non-prognostic secondary fields from addfld calls (sub def_fsech), 
and for T,U,VLBC (sub deflbc).

8/26/09: 
Have noticed a problem at bottom boundary of VN that is
related to number of processors used: bubbles show up esp with
64 procs. Astrid found that we were not doing mp_bndlons and
mp_bndlats calls for t,u,vlbc after reading the source history,
so these were added in rdsource.F.

9/1/09:
Fixed notorious O+ bug (OP, TE, NE not right above zp 0):
 - oplus.F:  Moved initialization of allocatable arrays 
             to before 1st latitude scan (its actually
             commented out in this version)
 - mpi.F:    Fix mp_bndlons_f3d to loop over j0-2,j1+2
             (also added iprint flag for debug purposes) 
Tuning for timegcm1.41.eqdnc:
  - cons.F     (prndtl)
  - dynamics.F (do not call xray)
  - gw_drag.F  (c(k))
  - mgw.F      (zhi, difk_gw)
  - settei.F   (Te=Te**2/7 first, then compare w/ TN)
There are still problems with lower boundary of UN,VN related
to number of processors, but it is small in magnitude, and
does not propogate upward. May need to modify mp_bndlons_f2d
similiar to mp_bndlons_f3d (see calls in rdsource.F).

Will do an svn commit, probably overwrite the v1.41 tag,
and overwrite $TGCMROOT/timegcm1.41.

9/2/09:
Did the above commit on 9/1, but not the overwrites.

Changes to subs mp_bndlons_f3d and mp_bndlons_f2d in mpi.F.
Now using jbeg,jend.  The f3d mod fixes OP,NE,TE above zp 0
(as indicated above), and the new f2d mod fixes lower boundary 
bubbles in U,V (especially visible in V), which changed w/ 
number of mpi tasks.

Fix rdsource.F to work for non-mpi jobs.

Modified scripts/Mknamelist to use steady-state files made
w/ this code and stored in $TGCMDATA:
FOSTER.timegcm1.41s.peqnc001.nc for single-res, and
FOSTER.timegcm1.41d.peqnc001.nc for double-res.
Also extended default run to a full day.

9/3/09:
Saved tag $SVN/timegcm/tags/timegcm1.41 from trunk.

10/6/09 (Astrid's mods to lbc, dated 9/28/09):
  input.F: shutdown if NCEP_REANALYSIS is set
    since this is 
    1. not supported and tested, 
    2. 6 hourly data which can be used with
    tidal perturbations, 
    3. ECMWF is available
   done in subroutine inp_lbc

  dynamics.F:
   new subroutine uvbgrd calculates Un,Vn at LBC for background only.
   subroutine uvbnd calculates Un,Vn at LBC for total field (background
   and perturbations). subroutine uvbnd is called the first 2 timesteps, because
   the Un, Vn, Z fields from tn-1 timsstep cannot be split into background and
   perturbations since the perturbations alone are not known.

   lbc.F: the additions are
     1. save the perturbations at tn and tn-1 for Un, VN
     2. save the background field for Z,Tn (Tn wouldn't have been necessary)
        in z_bgrd, t_bgrd
     3. save Un,Vn perturbations in u_prt, v_prt
     4. move perturbation part for Planetary waves,Kelvin waves and 2-day wave
     up before GSWM and Hough modes. These are added to the background since there is
     no explicit neutral wind component like for GSWM and the Hough modes
     therefore neutral winds at LB have to calculated using the subroutine 
     uvbnd/uvbgrd

   uvbnd.F: subroutine uvbnd calculates the Un,Vn at the LB for the background +
      perturbations
      new subroutine uvbgrd calculates Un,Vn at LBC for background only
      (exception is planetary, kelvin and 2 day wave which are included in the
      background). subroutine uvbnd is called the first 2 timesteps, because
      the Un, Vn, Z fields from tn-1 timsstep cannot be split into background and
      perturbations since the perturbations alone are not known.
      Note that in both routines the vertical advection term is NOT used
      anymore. This term has only a very minor influence on the results, and it
      would have been too elaborate to split the term into background and
      perturbation.

  btf: Add TIDEANN=1 to default namelist input file (see Mknamelist)

10/6/09: Doing svn commit, but not making new tag.
 
10/9/09: 
Fix bug in fft9.F in which ntrigs was left out of one of the two calls to 
FFT99B. This was pointed out by John Ritterer.

To Do:
 - Remove secfmag,secfgeo2d,secfmag2d,secfmagphr
 - Remove input_type

10/29/09:
Astrid's ecmwf.F using daily rather than 6-hrly data.
Somehow this got into 1-41 tag, but not 1.41 tag, and not the trunk.
This puts it in the trunk. This was svn revision 258.

Resurrect ncep reanalysis option, using daily averaged data instead of
6-hrly as lbc background, then adding gswm perturbations. See lbc.F
and input.F.  Source file ncep_rean.F replaces ncep_reanalysis.F.

11/4/09:
Realized that mp_updatephi was using an inordinate amount of time.
This routine passes global mag phim3d to non-root tasks after the dynamo.
Found that replacing mpi_send/mpi_recv with a simple collective mpi_bcast
call in this routine is *much* faster, apparently yielding speedup of 12% 
total run time for 64-proc bluefire run of double-res timegcm for a 1-day 
run, and a 25% speedup for a 5-day run at single-res!

11/4/09: svn commit. This does not yet contain per-sub dynamo timing.
11/5/09: Fix a couple of use-associations. "Final" commit was 267.

11/12/09: Check for ncep_reanalysis option combinations in sub
inp_lbc (input.F). ncep_reanalysis is treated in basically the
same was as ncep_ncfile and ecmwf_ncfile, i.e., they are background
lbc to which gswm perturbations may be added. But tideann and
planetary, kelvin, or 2-day waves are not allowed with ecmwf,
ncep, or ncep_rean. This change was put in $TGCMROOT/timegcm1.41/src/modsrc.

11/19/09: Add ncep_reanalysis to conditional on line 445 of lbc.F.

11/19/09: Set polar values 0 and nlatp1 in dynpot, magfield.F.

12/2/09 btf:
Check for unit conversion of u,v,z in sub get_ncep_data_daily, ncep_rean.F.
(formerly it was only in sub get_ncep_data, which used the 6-hrly data
rather than the daily).

1/5/10 btf:
Remove local declarations of ed13d,ed23d in sub threed, so the
dynamo module declarations of the same variables are used (dynamo.F).
As per Astrid (this was done in timegcm1-2dev7, but was not 
carried forward).

4/20/10 btf: adding Astrid's updates of apex_subs.F and sub sunloc_apex
from February, as follows (files copied from $TGCMROOT/timegcm1.41/src/modsrc):
2/4/10 maute:
file: apex_subs.F
update IGRF to igrf10 and igrf11 from website
http://www.ngdc.noaa.gov/IAGA/vmod/igrf.html

files: advance.F  apex_subs.F  magfield.F
  new subroutine sunloc_apex
change the the calculation of the
sun's location from dipole coordinates to apex. This is now conform with
the apex coordinate system (the difference is minor 1-3 degrees)

4/20/10 btf: Removed mpi_barrier call at end of timestep loop in advance.F.
This is not necessary, but does not appear to change performance. Also
removed unused time0_barrier, time1_barrier.

This code runs about 3.85 mins per simulated day with 32 procs on bluefire.

Saving this code as svn tag timegcm1.42
Changed tgcm_version to timegcm1.42 in params.F, and changed default
set model to timegcm1.42 in scripts/*.job.

4/21/10:
Added sub set_dynpot in rdsource.F to define dynpot for non-MPI runs
(as in tiegcm).

Added whole-array init to je1oDtrn and je2oDtrn in sub transf
of dynamo.F. Without this init, the code was crashing when
compiled without MPI (serial run).

8/23/10:
Realized that comp_dat.F had no NLEV conditional for 5-deg resolution
with upper boundary at zp +7. This meant that wn4s, wco2, etc were
all zero. Have asked Ray about this, but in the meantime, am explicitly
setting them to zero in this case. Also removed data statements in comp_dat.F 
for upper boundary at zp +5, since presumably, we will not be going back 
to that.

8/30/10:
Jia Yue pointed out that the local time xlth in gw_drag.F was being
calculated incorrectly as "xlth = uth+(-180.+(i-1)*5.)/15.", lines 1054 and 1101.
Altho xlth is not used (is commented out on line 1101), the name was changed
to slt, and set as follows:

          slt = fslt(slt,uth,glon(lon0+i-1),1)

where function fslt was added to util.F, and glon is use-assoc from init_module.
This should not change any results, because slt is not used.

10/19/10 Jia Yue: Always define time in sub bndry_planetary, lbc.F. This will
have an effect only if planetary waves are implemented by uncommenting code
in sub bndry_planetary (see comments in the source code).

10/27/10:
Commented out part about not allowing zatmos_ncfile at 2.5 deg resolution
in sub inp_lbc of input.F, because zatmos *is* allowed at either resolution.

11/1/10: Changed tgcm_version in params.F to "timegcm_trunk", and 
"set model = timegcm_trunk" in scripts/*.job. These should be changed to
the tag name when an svn tag is saved.

11/1/10: Committing to the trunk. Last commit was r319 on 4/21/10
