THE MEMORY WINDOW 135 also subscripted by indices i,k,j but without a fourth or fifth index since they are recalculated

for each prognostic variable to conserve memory. Additionally, some options require diffusive coefficients with spatial dependence. In this case, three dimensional arrays are used for diffusive coefficients which are also defined on cell faces.

A simplified schematic of the MW is given on the right side of Figure 11.2a by collapsing all indices except for the local latitude index “j”. Two important characteristics are indicated:

the MW contains jmw = 3 latitude rows and prognostic equations are integrated to update prognostic variables to time levelτ+1 starting on local row j = 2 and ending on local row j = 2. Rows on which prognostic equations are solved are indicated by an “‘x” and buffer rows which hold data one row away are indicated by a “circle” in the simplified schematic. An example of a larger MW is given in Figure 11.2b. In this example, the window holds jmw=6 rows and prognostic equations are solved for local rows j=2 throughj=5. In both windows, local rows given byj=1 andj= jmware needed as boundaries for the second order numerics.

Note that the MW is always symmetrical in that there are as many boundary rows to the north of the computed rows as to the south. Why aren’t prognostic equations solved on rows j=1 and j= jmw? Because indexing into arrays required by the 2nd order numerics would reach outside the MW (i.e. j<1 orj> jmw).

In general, the number of rows for which prognostic equations are solved within a MW is given by the size of the MW minus the number of northern and southern boundary or buffer rows “jbuf”. Equations for intermediate quantities such as density or fluxes on cell faces are solved on these extra rows. However, details about which quantities are solved and where they are defined on the grid system within the MW are not needed here and are covered more thoroughly in Section 22.2.4. The important point for now is that the number of computed rows “ncrows” where prognostic equations are solved within the MW is given by

ncrows= jmw−2∗jbu f (11.1)

and prognostic equations are solved on every row within the MW fromj=js calcthroughj=je calc where

js calc = 1+jbu f (11.2)

je calc = jmw− jbu f (11.3)

In the case of second order numerics, jbuf=1, the minimum sized window is jmw=3, and there is only one row on which prognostic equations are solved:js calc = je calc = 2. For a larger window where jmw> 3, prognostic equations are solved from MW rows js calc= 2 through

je calc= jmw−1.

For third and fourth order numerics, nearest neighbors of nearest neighbor cells are required which means a larger minimum sized MW is needed. An example is given in Figure 11.3a which is the minimum size required when executing with third or fourth order numerics. In this case,jbu f =2, the window holdsjmw=5 rows, and prognostic equations are solved from rows js calc = 3 through je calc = 3. As with the 2nd order MW, equations for intermediate quantities are solved on the buffer rows as long as the indexing does not reach outside of the MW.

11.3.2 Solving prognostic equations within the MW.

Refer to the example in Figure 11.4a. On disk (or ramdrive), there are jmt= 8 latitude rows arranged monotonically from south to north withjrow=1 representing the southernmost row

andjrow= jmtthe northernmost one. Notice also that the first and last rows are marked with starting and ending indices “jstask” and “jetask”. These define the limits of the processor’s task and the processor does not reference data outside of these limits. The other set of indices

“jscomp” and “jecomp” mark the starting and ending rows on which prognostic equations are solved (computed).

A minimum sized MW for 2nd order numerics is shown accessing data from latitude rows on disk (or ramdrive). Note that the global index jrow = 4 corresponds to local index j = 2 within the MW. The arrows pointing from the disk (or ramdrive) to the MW indicate thatτand τ−1 data in local MW rows j={1,2,3}are from disk latitude rows jrow={3,4,5}. Prognostic equations are solved on latitude row j=2 within the MW. The arrow pointing from the MW to the disk (or ramdrive) indicates that updated data at time levelτ+1 is written back to disk over thetau−1 slot. Note that the relation between local indexjand global index jrowis given by the offset jo f f in the figure. In general, for any local index j, the mapping to global index

jrowis

jrow= j+jo f f (11.4)

where offsetjo f f is simply how far the memory window has been moved northward from the latitude given by “jstask”.

In general, prognostic equations are solved on all jrows from “jscomp” to “jecomp” by a northward moving MW which solves a small group of rows (usually one) at a time. All steps are illustrated in Figure 11.4b. The first MW loads data from jrow= {1,2,3} and the offset is jo f f = 0. Prognostic equations are solved for local index j = 2 within the MW and written back to the jrow=2 slot on theτ−1 disk (or ramdrive). Then the MW is moved northward by the following operation. All data within the MW on row j=2 is copied to rowj=1 followed by all data on row j=3 being copied into rowj=2. This is explained more fully below. Then two time levels of disk data from global latitude row with indexjrow=4 are read from disk (or ramdrive) into MW row j=3 and the offset is bumped up by the number of computed rows which in this case is one. Therefore, jo f f = 1 for the second MW. After prognostic equations are solved for row j=2, the updated data is written back to the jrow=3 slot on theτ−1 disk (or ramdrive). The process continues until all rows from jscomp=2 to jecomp= jmt−1 have been updated toτ+1 values on theτ−1 disk (or ramdrive).

Figure 11.5 is the counterpart to Figure 11.4 for a fourth order window. Since, third and fourth order numerical schemes require two additional rows in the MW, the minimum size of the MW is jmw=5. Using five rows, the essential point is to calculate second order quantities on the three central rows j= {2,3,4}. Meridional differences of these second order quantities yield a fourth order result defined at the central row⁸ j=3. All schemes which use nearest neighbors of nearest neighbor cells require optionfourth order memory windowwhich is automatically enabled in filesize.hwhen any of the existing fourth order schemes are enabled.

Any new parameterization which requires the above 4th order differencing must also enable optionfourth order memory window. In contrast to the 2nd order MW, jo f f = −1 for the first MW and prognostic equations are solved for the central row which is j= 3. Note also, that

8When uni-tasking, since calculations proceed from south to north and the southern most latitude is land, the second order quantity can be set to zero atjrow=1. This may be utilized to reduce the minimum size of the MW tojmw=4which makes the MW assymetrical. This is no longer allowed because assymetrical MW’s lead to complications when multi-tasking. To make the assymetrical MW work when multi-tasking requires extra communication calls and the management of the MW becomes more complicated. For simplicity, the MW is required to be symmetric with the same number of buffer rows to the north and south of the computed rows. The price is 25% more memory for the symmetric MW which buys simplicity and generality.

11.3. THE MEMORY WINDOW 137 thej=1 row in the first MW hangs below the physical domain which starts at jrow=1. Data from row j= 2 is replicated in row j =1 to fill this first MW. As can be seen in Figure 11.5b, prognostic equations are still solved from rows jscomp= 2 to jecomp= jmt−1 as in the 2nd order MW case.

11.3.3 Moving the memory window

After prognostic equations have been integrated on rows within the MW and these rows have been written to disk (or ramdrive), the MW must be moved northward before computation can begin on the next group of rows. The movement takes place by copying quantities from the northernmost rows into the southernmost rows of the MW and then filling the remaining rows with data from disk (or ramdrive). The number of rows to be copied depends on the order of the MW. The ordering of the copies is important else data copied in one operation will be wiped out by the next copy. Which rows are to be copied is intimately tied into how arrays are dimensioned.

All arrays dimensioned within the MW must have their latitude dimension fall into one of the following four catagories:

dimension A(,,jmw) ! all cells within the MW dimension B(,,1:jmw-1) ! all cells except j=jmw dimension C(,,2:jmw) ! all cells except j=1

dimension D(,,2:jmw-1) ! all cells except j=1 and j=jmw

For further explanation and some examples of how array dimensions are determined, refer to Section 22.2.4. Assuming the above array structures, the copy part of the northward movement of the MW is demonstrated by the following segment of code which should be viewed in conjuction with Fig 11.6 to illustrate how the copy occurs for various array shapes and MW configurations. After prognostic equations have been solved on rows marked by “x”, τ+1 data from those rows is written to disk. Figure 11.7 illustrates the copy and move operations when option pressure gradient average is enabled. Note that tracers are computed on one row to the north of where baroclinic velocities are computed. The extra row is to allow for a four point average of pressure gradients when computing baroclinic velocities. When optionpressure gradient averageis enabled, only rows for which baroclinic equations are solved are written to disk (or ramdrive). Updated tracers on the extra row are copied into the next MW and get written to disk (or ramdrive) with baroclinic velocities from that MW.

for j=1,num rows to copy

jfrom = jmw - num rows to copy + j ! MW row to copy data from

jto = j ! MW row to copy data to

do k=1,km ! for arrays dimensioned (1:jmw)

do i=1,imt

A(i,k,jto) = A(i,k,jfrom) enddo

enddo

if (jfrom .le. jmw-1) then ! for arrays dimensioned (1:jmw-1) do k=1,km

do i=1,imt

B(i,k,jto) = B(i,k,jfrom) enddo

enddo endif

if (jto .ge. 2) then ! for arrays dimensioned (2:jmw) do k=1,km

do i=1,imt

C(i,k,jto) = C(i,k,jfrom) enddo

enddo endif

#if defined fourth order window ! for arrays dimensioned (2:jmw-1) if (jto .ge. 2 .and. jfrom .le. jmw-1) then

do k=1,km do i=1,imt

D(i,k,jto) = D(i,k,jfrom) enddo

enddo endif

#endif enddo

Note that the number of copied rows is typically⁹

num rows to copy=2∗jbu f (11.5)

which is the total number of buffer rows in the MW. The number of buffer rows required on the northern side and the southern side of the MW depends on the order of the MW (i.e. jbu f =n/2 for an n-th order MW). Note also that data in array “D” needs to be copied only when the MW is fourth order. After these copies have been made, the remaining rows in the MW from j=num rows to copy+1 through j= jmwmust be read from disk (or ramdrive) to fill up the MW before prognostic equations can be solved.

11.3.4 Questions and Answers

Here are some questions and answers about memory window and related issues:

• How much space does the memory window require?

There are 12 arrays required to hold three time levels of prognostic data for two hor-izontal velocity components and two tracers within the memory window. The arrays are dimensioned as A(imt,km,jmw). For the simplest options, the workspace part of the memory window must allow space for 3 advective velocities on the faces of T-cells, 3 advective velocities on the faces of U-cells, 3 temporary arrays for use as fluxes on cell faces, 1 array for density, and 2 arrays for land/sea masks. If it is assumed that all of these 12 arrays in the workspace part are dimensioned as above, then 50% of the space within the memory window is workspace and 50assumption when the memory window is fully opened (jmw=jmt for second order numerics). The percent of space taken by workspace arrays is significantly less when the window is minimum (jmw=3). But the minimum window is not the constraining configuration.

9When optionpressure gradient averageis enabled, the value ofnum rows to copymust be increased by one.