Forecasting Lab Guide to using the WRF Model at NC State - Part 2

From WolfWikis

Part II: The Extras

THIS DOCUMENT CONTAINS PART TWO OF THE ORIGINAL WRF WIKI GUIDE ("THE EXTRAS") THAT BECAME TOO LONG AND HAD TO BE DIVIDED UP...WE HOPE TO STREAMLINE THIS DOCUMENT A LITTLE BETTER AT SOME POINT, BUT FOR NOW AT LEAST IT IS HOPEFULLY FUNCTIONAL. -KMM, FEB 2008

Performing a nested WRF run

Assumptions: First, I am assuming that you have the WRF model installed as well as WRFSI and are able to open WRFSI without any problems. I also assume that you have been successful in performing a simulation using a single domain. If either of those statements are not true, then you will need to look elsewhere for more help before trying to perform a nested run. Here I will highlight the differences between running a single domain and running nested domains, but will skip over most of the procedure that is identical. The procedure for making a nested run is generally similar to a normal run so you need to begin by running WRFSI...

Running WRFSI

1. Create the outer domain:

After opening the program by typing wrf_tools, click on the domain selection button on the left hand side of the interface. Then you will want to either load your existing domain and place a nest within it, or create a new domain for your nested run. If you create a new domain you will first need to create the outer domain by dragging the cursor on the map, selecting the correct map projection and then clicking update map.

Next, you will see the domain which you have just created. This domain can be edited by clicking on one of the squares on the outer edge of the domain and dragging with the mouse. If you want to edit the grid in a more fine manner, to the right of the domain there is an option for the “fine scale editing mode”. If you select the option for “grid values” you can edit the number of horizontal grid points as well as the grid spacing.

At this point, you need to think about the grid spacing for your outer domain. If you will be performing a nested run with feedback on (i.e. data is passed between the grids) then you need to have an odd ratio between the outer domain and the inner domain(s). If you are leaving feedback off, this ratio can be odd or even. Either way, make sure that the ratio will be an integer.

As an example, the following grid spacings would be acceptable if you wanted feedback:

• 2 grids: 24 and 8km, or 21 and 7km
• 3 grids: 27, 9 and 3km

2. Create the inner nest(s)

After creating your outer domain and figuring out what grid spacing you are going to use for your nest(s), the next step is to create the nest(s). To the right of the map near the top, there are two tabs; MOAD domain and NEST domain. Click on the NEST domain tab to begin working on the nested domain.

First, you need to select the domain id. The first nest you create would have the id 2, a second nest would have the id 3, and so on for more nests. After selecting the id number, the parent id is automatically set as the domain that is outside of that inner domain. Next you can select the grid spacing ratio between the nested domain and its parent domain, keeping in mind that if you want feedback on this number must be odd. After those values are set, simply click on drag on the map to create the nested domain, like you did to create the outer domain. After creating the domain, you can edit it by changing the lower left and upper right grid values. Those numbers correspond to where the lower left and upper right grid points of your nested domain are located within the parent domain. Edit these numbers until you are satisfied with the location of the inner domain.

The procedure for adding more nested domains is identical to what you just did. To create a nested grid within the nest, simply go to domain id and click on d03. You will want to change the parent id to 2. After these changes, you would follow the same procedure as you did to create the first nested domain.

Finally, after you are done creating the domains click next. You then edit the vertical grid, the procedure here is similar to what you would do for a single domain run. For a nested run, you only specify the vertical grid dimensions once, and this will be used for all of the domains you are running. This means if you want to have very high vertical resolution on your inner-most domain, you will need to run all your domains with the same large number of vertical levels.

After clicking next again, you advance to the localization parms tab. Here you can see the values that exist for the domain configuration which you have already set up. Be sure that the num_domains variable is set to the total number of domains which you have created; i.e. 2 if you have one nested domain, 3 if you have 2 nested domains, etc. You may want to write down the values for DOMAIN_ORIGIN for the lower left and upper right grid points, because you will need these values later, or they can be found later on.

Next you will advance through the menus and localize the domain, which may take longer than it did for just 1 coarse domain so be patient.

The procedure for processing the initial data and interpolating the data is identical to what you do for a single domain. Before you interpolate the data, be sure that the following settings are correctly:

• Num_Active_Subnests: refers to the total number of nested domains you are using.
• Active_Subnests: should be the total number of domains in your model run. (note: if you are using more than one nested domain, this field must contain both subnest numbers. For ex, if you have 3 domains, this field should be "2,3", not just "3".)

When you are done with WRFSI, take a look at the output data, which is located in: /wrfsi/domains/<yourdomainname>/siprd

You should have wrf_real_input_d01 files for the duration of your model run, as you would for a single domain model run. You should also have a wrf_real_input_d0* file just at the initial time for your model run, where the * refers to the number of the domain. If you have one nested domain, you will just have one wrf_real_input_d02 file at the initial time; if you have 2 nested domains, then you should have wrf_real_input_d02 and wrf_real_input_d03 files, and so on for however many domains you use.

• • If you do not have the proper input files, check the num_domains variable in wrfsi and make sure the number there is the number of grids in your model run.

Copy or move these initial condition files to the WRFV2/test/em_real directory, and next these initial condition files need to be processed using real.exe.

WRF Procedure

1. Recompile WRF:

Depending on what option you chose when you initially compiled wrf you may or may not need to recompile the model but you may as well because if you did not select the option which enables nesting the model run will crash. Go to your WRFV2 directory, type compile, and select option 5. Be sure to type “add pgi” into the terminal window before you compile. To compile, type “compile em_real”

2. Preparing and running real.exe for the nested domain(s)

Next, go to the test/em_real directory. You will need to run real.exe as before, but the procedure for running it with nested domains is somewhat tedious. You will need to run real.exe twice if you have one nested domain and 3 times if you have 2 nested domains. Each time you run real.exe you are just processing the initial condition files for one of the domains at a time. You will run real.exe for the domain with the smallest grid spacing first, and then end by running it for the coarsest domain.

2a. Rename some files

The procedure for running real.exe is the same as before, however each time you run it you need to be sure that the files and variables are set for the domain you are processing. Since real.exe will be looking for the wrf_real_input_d01 file, you need to rename the file you are trying to process as d01.

First, rename the actual wrf_real_input_d01 file as something else (i.e. original_wrf_real_input_d01) so you do not accidentally overwrite it. Next, rename the wrf_real_input_d02 file (if you have 1 nested domain) or the wrf_real_input_d03 file (if you have 2 nested domains) as wrf_real_input_d01. This way, real.exe will process the domain with the smallest grid spacing first.

2b. Edit the namelist.input file for the domain you are processing

Next, you need to edit the namelist.input file. Since real.exe processes 1 domain at a time, you just need to edit the first column of numbers. Therefore, make sure that the “input from file” row of the namelist is set to true for only the first column. At this point, you need to edit the namelist.input file for the innermost domain. Most of the values in the file do not need to be edited, with the following exceptions:

• Run days, hours, minutes and seconds: set all these values to 0, since you are only processing the files for the nested domains at the initial time.
• Start time and end time: be sure to set the end date/time to the same as the start date/time, with both being the start time of your model run. The reason is you are only processing the files for the nested domains at the initial time.
• max_dom: this number refers to the number of domains you have, and at this time since you are only processing one domain should be set to 1.
• s_we, e_we, s_sn, e_sn:
• s_we and s_sn: refers to the start point of your grid in the west/east direction and north/south direction, respectively. Here both should be set to 1.
• e_we and e_sn: refers to the end points of the grid you are processing. These values were listed in wrfsi, or can be found by typing the following into a separate terminal window:

 ncdump -h wrf_real_input_em.d0*

where * refers to the number of the domain. The grid dimensions can be found near the end of the dump: WEST-EAST_GRID_DIMENSION and SOUTH-NORTH_GRID_DIMENSION. These represent the end points of the grid, and use these values to set e_we and e_sn:

• e_we: should be set to the value found after: “ WEST-EAST_GRID_DIMENSION”, or the value found while running wrfsi.
• e_sn: should be set to the value found after: “ SOUTH-NORTH_GRID_DIMENSION”, or the value found while running wrfsi.
• dx and dy: be sure to set these values to the grid spacing you chose while running wrfsi, noting that this number should be in meters.

Note that other values in the column remain the same, and you do not need to change values such as grid id, level, i_parent_start, etc. since we are treating the domain as if it is the only domain.

2c. Run real.exe

Now you should be all set to run real.exe, so simply type real.exe and see what happens (be sure you are on a compute node). If it runs successfully you will get a wrfinput_d01 file. If it does not work successfully, then you probably didn't set the grid dimensions properly and you should be sure to set e_we and e_sn correctly.

2d. Rename some files (again), save your namelist

Next, you need to rename the wrf_real_input_em.d01 file back to its original name (either d02 or d03), and also rename the wrfinput_d01 file that you just got to wrfinput_d0*, where * refers to the number of the domain that you just processed.

In order to run real.exe again, you will need to change the values in the namelist.input file for the next domain. You will probably want to save the namelist.input file you created for the nested domain so you can have it if you ever need to rerun real.exe. Rename it something like namelist*.input, where * refers to the number of the domain that you set the namelist values for.

2e. Run real.exe for the next nested domain(s) (if you have more than 1 nested domain)

If you have another nested domain, you need to repeat the above procedure for that domain.

3. Run real.exe for your outermost domain

Finally, you need to run real.exe for the last time in order to process the initial condition files for your outermost domain. Be sure to change the name of the wrf input file for the outermost domain back to its original name before running real.exe again. In addition, you will need to make a few changes to the namelist.input file, as outlined below:

• run days, hours, etc: now you are running real.exe for the duration of your model run, so set these values to the proper length of your model run.
• end year, month, etc: again, now you need to set these values to correspond to the proper end time of your model run.
• change grid spacing and grid dimensions to properly reflect the values of your outermost grid.

After running real.exe for the outermost domain, you should get wrfinput_d01 and wrfbdy_d01 files. This time you do not need to rename them!

4.Setting up the namelist to submit your model run

Now you need to edit the namelist.input file one last time. This time you will be making changes to either 2 or 3 columns depending on the number of nested domains you are using. Be sure to save the namelist.input file you last used for the outermost domain before you start editing it so you have it in case you need it later.

There are a lot of values you need to edit, so be sure to make the following changes:

• start time/end time: keep in mind you can start and end the simulations on individual domains at different times if you would like.
• input_from_file: set column 2 to true if you have one nest, and set column 2 and 3 to true if you have 2 nested domains.
• history_interval: you probably want to set this value to 180 for every grid so you get input every 3 hours, but you could use other values as well.
• frames_per_outfile: set this to 1 for each grid so you get an output file at each output interval.
• max_dom: be sure to set this value to the total number of domains that you are using (2 if you have one nested domain, 3 if you are using 2 nested domains)
• e_we: the value in these columns corresponds to the end point in the east-west direction for each grid. The value here for each grid will be the same as the e_we value that you used when you were running real.exe for that specific grid. If you forgot to save those namelists, you can always use the ncdump -h command as earlier and look at WEST-EAST_GRID_DIMENSION.
• e_sn: same as e_we, except the end point in the north-south direction for each grid. Follow instructions above for e_we, and use the value found after SOUTH-NORTH_GRID_DIMENSION.
• e_vert: be sure to change this value, if necessary, in the column(s) corresponding to your nested domain(s).
• dx and dy: change these values to correspond to your domains. (value in meters)
• i_parent_start and j_parent_start: for the outer domain, these values should both be 0. For your nested domain(s) these values, can be found by using the ncdump -h command and looking at i_parent_start and j_parent_start.
• parent_grid_ratio and parent_time_step_ratio: these values should be the same and correspond to the ratio between the outer grid and the nested grid. Example: If you had a 27 km outer grid and 9 km nested grid, then the value in the second column would be 3. The model needs to use a smaller time step for the inner grid in order to satisfy CFL criteria.
• feedback: set to 0 for no feedback, or 1 if you want feedback.
• physics options: you can use different physics options on the different domains if desired. For example, if you had a small enough grid spacing on the nested domain you could choose to not use cumulus parameterization. Or, you can set the values the same for each domain.

5. Submit the model run to the cluster

The procedure for submitting the nested WRF run to the cluster is the same as for submitting a regular model run. You will want to request more processors due to the greater number of grid points that you will be making calculations on.

That's it! Good luck. The following websites could be useful if you need more information that wasn't provided here:

WRF User Guide:
http://www.mmm.ucar.edu/wrf/users/docs/user_guide/contents.html

WRFSI Information:
http://wrfsi.noaa.gov/gui/users_guide/V2/users_guide.html

Vortex-tracking Moving Nest

This section describes implementation of the automatic moving nest with version 2.1.

Limitations include...

• only a 3:1 nest ratio can be used
• available for EM core only
• inner-most nest tracks vortex using a “a mid-level vortex following algorthm”

1. Compiling with moving nest option

First, type “configure” and choose your option, as described above. Then add the -DMOVE_NESTS and -DVORTEX_CENTER flags to ARCHFLAGS in configure.wrf. My ARCHFLAGS looks like this...

ARCHFLAGS       =       -DDEREF_KLUDGE -DIO_DEREF_KLUDGE -DGRIB1 -DINTIO -DWRF_RSL_IO -DRSL -DDM_PARALLEL\
                        -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=$(RWORDSIZE) -DLWORDSIZE=4 -DNETCDF \
                        -DTRIEDNTRUE   \
                        -DMOVE_NESTS -DVORTEX_CENTER \
                        -DLIMIT_ARGS

The hard-coded limit for the number of moves is 50. To change this, modify “max_moves” in configure.wrf. Then compile. 2. Setting up domains

The procedure for setting up a moving nested run is the same as a normal nested run, as described above, but with the following considerations. When setting up the inner, moving nest, use a large outer domain in order to prevent the inner nest from bumping into the edge of the mother domain. The closer your moving domain is to the boundaries, the more likely it is to lose your initial vortex and chase down a smaller perturbation associated with lateral boundary effects. Also, the moving nest is only allowed to get within a certain number of gridpoints of the parent.

3. How the nest moves

The moving nest only moves if the vortex gets 3 grid boxes or more away from the center of the nest. It checks to see if it needs to move every 3 timesteps of the inner grid or every timestep of the outer grid. The moving nest will only shift over 1 parent gridpoint (or 3 inner nest gridpoints) at a time. To see if the nested grid is moving correctly or bumping into the the outermost domain's edge, check the rsl.error files.

One of the more difficult parts of getting the moving nest to work correctly is getting it to track the right feature. Placing the moving nest too close to the outermost, mother domain not only limits the nest's ability to move, but also makes it more likely that the nest will track a perturbation associated with the boundaries instead of the actual feature you want to model.

When using multiple nests, remember that only the innermost nest actually tracks the vortex, and all nests except for the outermost, mother domain can move. For example, in a situation with 3 domains, d01 cannot move and only d03 tracks the vortex. The middle domain, d02, moves in response to the innermost nest getting within a certain number of gridpoints of its boundary. The middle nest will just be “bumped” along by the innermost nest.

The exact logic of how the vortex tracking algorithm works, as well as how all the movable nests move is in the FORTRAN code:

WRFV2/share/mediation_nest_move.f

For more information... http://www.mmm.ucar.edu/wrf/users/wrfv2/moving_nest.html

Adding Output Variables to WRF

Disclaimer: I am assuming that if you are interested in adding variables to WRF output you have already completed successful WRF model runs and therefore do not need any information on how to actually run the model. Here you will find instructions just on how to get more output variables; further instructions on how to set up WRF can be found in other documents.

Introduction:

Although WRF already outputs most meteorological variables of interest, there are times when a user may be interested in looking at extra variables that may not be included in the model's “standard” output. These variables may be specific to a certain physical parameterization scheme, or of other interest. The manner in which WRF was designed allows extra variables to be added to the output fairly painlessly, and only involves a few steps.

Edit the Registry file

• cd to the WRF registry directory. Example:

 /share/youruserid/WRFV2/Registry

Open the file Registry.EM

Overall, the Registry.EM file is fairly well commented and should be fairly easy to read. The file basically consists of the different variables that are used in WRF, and also contains information on these variables. Line 63 lists the information that is included on each variable; for this case, we are most interested in the column <IO>. Listed for each variable under this category there are 3 main letters that may or may not be present:

• r: indicates that the variable is written to WRF restart files, which can be used to restart the model if the model crashes at a certain amount of time through the model run.
• i: indicates that the variable is an input variable
• h: indicates that the variable is an output variable

It may seem to easy, but all that you need to do to get a variable output by the model is add the letter “h” to the <IO> column! The only challenge is determining which variables you want to output. Throughout the Registry.EM file there are comments indicating which physics packages contain certain variables, or what the variables are generally used for. Pay attention to these labels and you should be able to find what you are looking for. Also, pay attention to the units of the variable, which are listed on the far right column of the document. Note that the units may not be exactly what you would anticipate, since the values are often multiplied by other factors.

• Recompile the model
• After editing the Registry.EM file, the model needs to be recompiled. Note that the Registry.EM file is the only file that needs to be edited; after recompiling, the Registry file is automatically updated with the changes you made.
• To recompile the model, move up one directory from the Registry directory. Example:

 /share/youruserid/WRFV2/

• Before recompiling, be sure you are logged in on the login node, and also be sure to type “add pgi” into the terminal window in order to be sure the proper software is sourced.
• After completing the previous 2 steps, type “clean” into the terminal window to remove the previous files that were installed.
• Type “configure” and select the compilation option that you would like.
• Type “compile em_real” and the compilation process will begin. It should take about 20 minutes for the model to compile, after which you should be able to rerun the model and get the new output variables.

Make sure the new variables are there

• After the model has finished running, there should be a selection of wrfout files in the following directory:

/WRFV2/test/em_real

• In order to check the variables that are located in the wrfout file, type the following command:

 ncdump -h wrfout* > output_file

Where wrfout* is the actual name of one of the output files, and output_file is the name of the text file where the information will be written. Note that if you do not type the > output_file part of the previous command, the output will be placed in the terminal window, but it is easier to find what you are looking for if the output is placed in a text file.

• Next, open the output_file and search for the variables that you added as output; if the model did indeed output them, then they should be listed! If not, you may need to go back and repeat the procedure. Also, note that the information listed will contain the units of the variable as well as the variable's name, both of which could be useful!

Next up: viewing the new output variables in GEMPAK, which requires editing the wrf2gem conversion code! See next section...

Changing wrf2gem Output Variables

Last section the procedure for obtaining more output variables from WRF was described. Unfortunately, these variables are of no use unless you can plot and look at them. Most likely, you have been using a script called wrf2gem_gl.csh to convert your WRF output to Gempak format (let's all thank Dr. Mike Brennan and Dr. Gary Lackmann in our heads right now, as well as others involved in the creation of this code!).

The c-shell script wrf2gem.csh actually is an interface that runs an executable Fortran program, wrf2gempak. The c-shell script, if you look at line 28, copies the executable Fortran program from the following directory:

/lockers/PAMS/Volume1/gary/MEA717

In order to get wrf2gem to output more variables, it is this Fortran code that must be edited, and not the c-shell script. The steps involved in this procedure follow.

1. Copy the necessary files to your directory

To begin, create a directory called wrf2gempak in your locker space. Move into that new folder, and then copy some files into that directory from the following directory:

/lockers/PAMS/Volume1/kahill/wrf2gempak

To copy the files, from within your wrf2gempak directory type the following command:

 cp /lockers/PAMS/Volume1/kahill/wrf2gempak/* .

If you take a look at the files within this directory, you will notice the familiar wrf2gem_gl.csh script, as well as the Fortran program wrf2gempak. What we will be doing is editing the Fortran program, wrf2gempak.F. After editing this file, we will compile it, creating a new wrf2gempak executable that will convert the new variables.

2. Edit wrf2gempak.F

Open up wrf2gempak.F. Adding the new output variables is fairly straightforward, but changes do need to be made at various locations in this Fortran program. Be sure to make changes to all of the following locations and things should work! For examples, here I will describe how to add the variable “wrf” to the Fortran program. Note that the line numbers I am referencing here are the line numbers from the original Fortran code; if you add lines then obviously the line numbers will not match up exactly! Also, you may want to rename and save the original version of wrf2gempak.F so if your changes do not end up working you can have a fresh copy to look at.

• Lines 43-66

These lines are necessary to allocate the variables that will be converted. Essentially, your new variable just needs to be added to this list. In order to add it, you must know the dimensions of the variable. For example, if the new variable is 2-dimensional (variable is at only one vertical level) then the following line of code would be added at line 67:

 &                     wrf(:,:)

If the new variable were 3-dimensional (i.e. on different pressure or height levels) then the following line would be added:

 & 		wrf(:,:,:)

Of course, be sure to line up the “&” character (which just means that the line is a continuation of the previous line) to the correct column (column 6).

• Lines 137-161

These lines allocate the size of the variables for the Gempak output files. Again, at the end of the list add your variable(s), being sure to replicate the layout of the previously written lines. Again, be sure to pay attention to the dimensions. Add either of the following lines depending on the dimension:

 2-dimensional: 		allocate(wrf(ixx,iyy))
 3-dimensional: 		allocate(wrf(ixx,iyy,izz))

• Lines 205 to 223

Here the variables are read from the wrfout files using a subroutine located at the bottom of the Fortran program. These lines were added by Mike Brennan to output some extra variables that he wanted to see. Again, we need to just add a line of code for the new variable. Note that lines 212-225 consist of the variables that are output if the flag for extra output in the wrf2gem_gl.csh script is set to yes. If you want your new variable to be output regardless of whether that flag is yes or not (which is probably what you want), place the new line of code outside of that “if” statement (i.e. add a line after 211 for your variable).

 2-dimensional:		get_variable2d(cdfid, 'WRF', ixx, iyy, time, WRF)
 3-dimensional: 		get_variable2d(cdfid, 'WRF', ixx, iyy, izz, time, WRF)

Pay close attention to the stuff inside the parentheses here. Note that the expression within the quotations (i.e. 'WRF') is the name of the variable as output by the model. The last argument within the parentheses is the array that these values will be placed in. Most of the time, both these may be the same name. But, at times you might want to read the variable into an array with a different name. Line 222 is an example of this:

 get_variable2d(cdfid, 'LH', ixx, iyy, time, lhflux)

Here, 'LH' is the name of the output from the model, and these values are read into the lhflux array. In this case, the lines of code that were previously added would need to be added for lhflux, since that is the array being used in the Fortran program.

• Lines 261-361

These lines are where the output is written to the Gempak file. Before, we have not paid attention to where the given variable was located (i.e. at the surface, or at a pressure level). Within these lines there are variables that are written at the surface, at pressure levels or at height levels.

264-275: variables written at surface

280-287: variables written at 2-meters

291-294: variables written at 10-meters

297-298: variable written at surface

302-321: variables written at surface, if extra output flag set to yes.

325-362: variables written at different pressure levels.

Depending on where your variable is located, you should add a line that mimicks the correct write statement. Although, it really doesn't matter what location you write the variable at, as long as you remember where it is when looking at your Gempak output!

As a few examples, here are different lines of code depending on where you wanted to write the variable. Again, be sure to have the spacing identical to the lines already there.

Surface: add the following lines after line 275:

                call gd_wdat(gdfln1, wrf, ixx, iyy, hdrsz1,
     &          gem_time, gemlev, 0, 'WRF', .TRUE., iret)

Here, note that 'WRF' found in the second line will be the name of the variable in Gempak. You can change that to anything you want.

At a height level: add the following lines after line 294

       gemlev(1) = *
       call gd_wdat(gdfln1, wrf, ixx, iyy, hdrsz1,
              gem_time, gemlev, 3, 'WRF', .TRUE., iret)

Here, the * that I wrote after gemlev(1) refers to the height level where you want the variable to be placed (typically either 2 or 10 meters).

On pressure coordinates: add after 344 the following line

       call gd_wdat(gdfln1, wrf(:,:,i), ixx, iyy, hdrsz1, 
              gem_time, gemlev, 1280724037, 'WRF', .TRUE., iret)

• Lines 365-387

These lines deallocate the array that was previously allocated. Add the following after line 387:

       deallocate(wrf)

3. Compile the new code

Next, you need to compile this new code and create the new executable. Be sure you are on the login node and type “add pgi” before taking the following steps.

i.) Type “make clean” to remove the previously created files (note that you may want to rename and save the old one before doing this).

ii.)Type “make” to compile the code. If this works, you will see the file wrf2gempak in the directory. If it doesn't work, you should see the lines where there are errors in wrf2gempak.F. Carefully try and make corrections, and repeat the previous steps and try again.

4. Edit wrf2gem_gl.csh

Lastly, the c-shell script needs to be modified. Line 28 of the code copies the wrf2gempak executable file to the directory where the script is being run. Change the path in that line to reflect the wrf2gempak directory that you created. Now when you run the c-shell script, your Fortran executable file will be copied into that directory and run instead of the original one.

Note: If you see that the additional variables are written to the eta-level files but not the pressure-level files, then you have exceeded the gempak variable threshold (something that can only be fixed by recompiling gempak.) This is most likely to occur if you have added several three-dimensional variables. A workaround involves following the same types of steps as you would to view hydrometeor fields, discussed in section 10

Viewing hydrometeor fields

In order to view the hydrometeor fields output by WRF, another script must be run to interpolate the hydrometeor fields from eta vertical coordinates to pressure coordinates. Scripts to do this (which will have to be modified based on your run name and other specifics) are located in: /lockers/PAMS/Volume1/kmmahon2/wrf_files_use/hydrometeor_script_km.csh (for individual output files)

/lockers/PAMS/Volume1/kmmahon2/wrf_files_use/hydrometeor_script_LOOP_km.csh (to loop over all output files)

If not using the scripts, the essential steps are as follows: 1. Make an empty gempak file for the hydrometeor fields (use gdcfil, and let cpyfil = any eta-level file that you got from wrf2gempak.) 2. Use gdmod to move all of the hydrometeor fields into the empty file. You will need to copy qcld, qice, qrai, qsno, qgra at all levels, and you will need hght and pres at the surface (gvcord=none, glev=0). 3. Interpolate from eta coordinates to pressure coordinates (can use eta_to_pres_loop.csh for this)

Obtaining sea-surface temperature data

0.083° SST

A daily global 1/12° SST analysis is available from NCEP dating back 60 days. A description of this dataset is available at http://polar.ncep.noaa.gov/sst/ophi/

The data is available for download in grib form from:

ftp://polar.ncep.noaa.gov/pub/history/sst/ophi

The data can be converted to GEMPAK format for inspection as described in the next section.

0.5° SST

A daily SST analysis at 0.5° is available from NCEP going back through Feb. 11, 2001. A description of the data is available at: http://polar.ncep.noaa.gov/sst/oper/Welcome.html

To download the data, go here:

ftp://polar.ncep.noaa.gov/pub/history/sst

Save the file for the date(s) you want. The files are in grib format, to convert them to GEMPAK format using the following command:

 dcgrib2 YYYYMMDD_sst.gem < rtg_sst_grb_0.5.YYYYMMDD

Where YYMMDD_sst.gem is the GEMPAK file name, and rtg_sst_grb_0.5.YYYYMMDD is the name of the raw SST data file. It is a good idea to look at these data in GEMPAK to make sure the field is reasonable!

1° SST data

Prior to 11 Feb. 2001, a weekly 1.0-degree SST analysis is available here:

ftp://ftp.emc.ncep.noaa.gov/cmb/sst/oisst_v2/

The data are stored in .tar files by year going back to 1981. Download a yearly file, and untar it. Then unzip the file from the particular week you want. Converting these files requires a bit more work than the 0.5-degree ones, so you will need to take the following steps:

• Copy the file sst.f from this directory: /fs/gale1/wdata/mjb/sst/1.0degree (currently not available!)
• Edit the filename for your input file in this line:

 open(11,file='oisst.20001213',form='unformatted')

• Edit the output file name here:

 file2 = '001213_rsst.gem'

• Edit the GEMPAK time for the grid here:

 gdat(1)   = '001213/0000'

• Compile the program using the following command:

 f77 -o sst.exe sst.f $GEMLIB/gemlib.a

• Copy the script gdcfil.csh from the /fs/gale1/wdata/mjb/sst/1.0degree directory
• Edit the name of the output file to match what you specified in sst.f
• Run the script by typing gdcfil.csh, this creates a GEMPAK grid file with the proper dimensions for the SST data
• Run sst.exe, and the SST data should be written to the GEMPAK grid file you just created.

Then, you can convert the GEMPAK file to grib format and use it with the WRFSI.

OR

You could just get the 1 degree SST files already in GRIB format in a weekly format from: http://nomad3.ncep.noaa.gov/pub/sst/weekly/

Just in case the above link changes: Another way to get to this data is through the NOMADS site—the same place where you can get NARR data. http://nomads.ncdc.noaa.gov/data.php?name=access

Scroll down from the NARR section and click on the “http” link in the “Reynolds SST O/I v2.0” dataset row. From there, click on the “weekly” directory. In the “weekly” directory, you can find weekly SST data from 1981 to the present. Right click to save the files to your computer.

Modifying WRF input files

read_wrf_nc.f

read_wrf_nc.f is a WRF utility program and can be downloaded from... http://www.mmm.ucar.edu/wrf/users/utilities/read_wrf_nc.htm

This FORTRAN 90 program can read and modify the NetCDF files wrfinput_d01 and wrfbdy_d01.

First, compile using this command...

 pgf90 read_wrf_nc.f -L/opt/netcdf-3.6.0-p1/lib -lnetcdf 
 -lm -I/opt/netcdf-3.6.0-p1/include -Mfree -o read_wrf_nc

Run with....

 read_wrf_nc [wrfinput file name] [-options]

Use the -h option, see the above webpage, or see the header for a full list of options, but here are some of the most useful ones....

-m
Print full list of variables in file, as well as min, max, and units

-EditData VAR
Use the IF's starting starting on line 1000 of the FORTRAN code to modify any variable in the file (when modifying the SST, you must modify the variables SST and TSK)

Running WRF on the Henry2 cluster

If you’ve already done everything through the wrf_prep.sh stage (i.e. you’ve run real.exe and have all of your input files) on bigdog, then all you have to do is…

1. Unzip on /share3/ and configure and compile as before. Before compiling be sure to type “add pgi”

• Some compiling specifics: The latest version of NETCDF needs to be sourced in your aliases.csh file as the following:

 setenv NETCDF /usr/local/apps/netcdf-3.6.0-p1/pgi/linux86/6.0

• The above version works for the latest version of WRF (V2.1), but if you are using V2.0.3.1, you will need to source:

 setenv NETCDF /usr/local/apps/netcdf-3.6.0-p1/pgi/linux86/5.2

2. Move all of your input files to the appropriate places in your new directories on henry2 (you’ll need all your wrf_real_input_em* files, your wrfbdy_01 and wrfinput_01 files, as well as your namelist.)

3. Finally, run using a script like this:

 ******************************************************
 #! /bin/csh 
 # 
 #BSUB -o standard_output 
 #BSUB -e standard_error 
 # 
 source /usr/local/lsf/conf/cshrc.lsf 
 cd /share3/kmmahon2/WRFV2/test/em_real/ 
 mpiexec /share3/kmmahon2/WRFV2/test/em_real/wrf.exe 
 ******************************************************

And you submit the script with this command:

 bsub -W 18:00 -n 16 < wrf_kmm.csh

After the -W you specify the amount of clock time in hours for your run, and after -n you specify the number of processors you want. The length of time you can request for a run on henry2 is inversely proportional to the number of processors you request. In other words, if you request more processors, you will be restricted to a shorter run time, so try and choose the optimum number for both!

Helpful commands on henry2…

 bkill job#  		- to delete a job
 bsub    			- to submit a job
 bjobs or bpeek  	- to see job status
 bjobs –u all –a    	- to see everyone’s job status 

For those running jobs specifically on the henry2 cluster, the following links provide information that may be helpful; these were contributed by Dr. Matt Parker and his graduate students:
http://www.meas.ncsu.edu/storms/documents/WRF_on_henry2.pdf
http://www.meas.ncsu.edu/storms/documents/troubleshooting_wrf.pdf
http://www.meas.ncsu.edu/storms/documents/WRF_restarts.txt

General troubleshooting

1. In the wrfsi stages, log files of the errors are found in: /WRFV2/wrfsi/domains/yourdomainname/log/
2. If wrf.exe and/or ideal.exe are missing, then your compilation failed.
   • append > messages to your compile command so that you can quickly see any coding errors… go in and fix them
   • if your code syntax is okay but it’s still not compiling, especially if you’re getting a library error, do a full ./ clean –a and a fresh configure & compile
3. Once the model compiles, if the run stops pre-maturely…
   • Check rsl.out.0000 to determine how long the model ran. If it didn’t run at all, you may have a bad path name or filename in your batch script, or a file that the model needs may be missing. Are your paths correct? Do you have gribmap.txt in your run directory (run starts but you get an “error reading” message)? Etc.
     • If you’ve submitted a .csh or .sh script, also look at the scriptname.eXXXX or scriptname.oXXXX created in the directory you’re running in, in order to shed some light on what might be going wrong.
   • Did your run hit the wallclock limit? If so, use restart files to proceed.
   • If the model didn’t appear to fail for either of these reasons, check your output at the last available time before the model stopped. Is everything stable and physical looking?
   • See if the model violated the CFL criterion… a quick check is to do a grep cfl *err* in your run directory.
   • If something went wrong very early on, check the output from your initial time. Is it physical? Is it what you wanted?
   • If everything looks physical but the model violated the CFL criterion, decreasing the timestep may help. If things look unphysical very early on (i.e. the initial time), then something went wrong in the initialization. Check your input soundings, etc.
4. Once the model has completed a simulation, do some quick quality checks…
   • Does your output look like what you expected? Check the environmental temperature and humidity profiles, check the environmental wind profile, and check whatever initial perturbation you added. Are the structures, positions, and magnitudes correct?
   • If not, possible causes are a) your wrf2grads (or wrf2gem) conversion was pointing to the wrong files (i.e. you’re actually looking at output from a different run): check path and filenames; or, b) your input sounding and/or code for the initial perturbation had errors… double check, correct, and rerun the simulation.

Other random, common problems

• Bad Data:
  • Does your data have hght, temp, and moisture variables at all of the necessary levels?
  • Data issues to consider:
    • Eta 212 does not have moisture variables above 300mb, and WRF will not run unless you have moisture data at all levels.
    • NARR does not have RELH fields and will need to be calculated (this can be done fairly easily)
    • Some model data obtained from NARR may only have surface data in it (e.g. Eta 215) – be sure your files have all necessary data at all necessary levels!

• “Broken pipe” error?
  • You may need to request more memory from the processors. Try increasing your number of processors (after first making sure that the nio_tasks_per_group number matches that in your submit script.)

• Run quits for no reason, and with no error messages?
  • Not sure what the problem is, but try running it on the henry2 cluster instead (or vice-versa)!

Sample scripts

Example: wrf_prep.sh

****************************************************************
#! /bin/csh 
#$ -cwd 
#
set runname = mar31_4km_exp
set yourid = `whoami`
set wrfpath = /share/$yourid/WRFV2/test/em_real
cd $wrfpath
#
#$ -pe mpich 4
echo “ Running on $NSLOTS processors.”
#

setenv P4_GLOBMEMSIZE 536870912
unlimit

/usr/lib/mpich/pgi/bin/mpirun –np $NSLOTS –machinefile $TMPDIR/machines $wrfpath/real.exe

****************************************************************

Example: wrf_run.sh

****************************************************************
#! /bin/csh 
#$ -cwd 
#
set runname = mar31_4km_exp
set yourid = `whoami`
set wrfpath = /share/$yourid/WRFV2/test/em_real
cd $wrfpath
#
#$ -pe mpich 16
echo “ Running on $NSLOTS processors.”
#

setenv P4_GLOBMEMSIZE 536870912
unlimit

/usr/lib/mpich/pgi/bin/mpirun –np $NSLOTS –machinefile $TMPDIR/machines $wrfpath/wrf.exe

****************************************************************

North American Regional Reanalysis (NARR) usage

a. Source

A valuable source of data with which to initialize WRF for historical events is the North American Regional Reanalysis (NARR) dataset, obtainable from the NCDC or NCAR Nomads servers:

http://nomads.ncdc.noaa.gov/data-access.html#weather_datasets

http://dss.ucar.edu/pub/narr/

For additional documentation, see the NARR homepage:

http://wwwt.emc.ncep.noaa.gov/mmb/rreanl/index.html

When downloading, obtain the narr-a files, rather than the narr-b files. The difference has to do with Grads formatting, but the –b files don’t work so well with WRFSI. There are some other issues with this excellent dataset, perhaps most importantly is that the winds are stored as north-relative, despite the use of a Lambert Conic Conformal (LCC) grid structure. Normally, one would store the grid-relative winds on a conformal map projection, and only store north-relative winds on lat/long type grids such as cylindrical equidistant or mercator projections. For small domains, this matters little. But for synoptic-scale domains, there is significant distortion towards the edges of the grid, where grid rows depart significantly from latitude lines.

The consequence for WRF is that when degribbing the NARR data in WRFSI, it assumes that the winds are grid relative, since it is working with a conformal projection. Therefore, special steps must be taken in order to properly degrib the winds for initial and lateral boundary conditions derived from NARR data:

b.) Wind fix

1.) *Before compiling WRFSI*, move to your wrfsi/ directory and copy the tar file containing the corrected code:

 cp  /lockers/PAMS/Volume1/gary/wrf/narr_si.tar .

2.) Untar this file there ( tar –xvf narr_si.tar )

This step takes care of all modifications needed for Vtables, etc.

3.) Compile WRFSI as usual

c.) Running WRFSI

Then, when you are running WRFSI, you will need to run for NARR as before, but you will need to do a second run for NARRSFC, using the pull-down menu in the grib_prep stage. Note that this may not work if you have extraneous files in the directory in which the narr grib files reside! Note: you will also likely have to run with the RUC LSM (sf_surface_physics=3 in namelist.)

Using the NCEP/NCAR Reanalysis Dataset (NNRP)

a. Source

Another useful initialization dataset is the NCEP/NCAR Reanalysis. This dataset has 2.5 degree resolution and analysis every 6 hours. Because of the coarse resolution, this dataset is best used for synoptic scale simulations. The most recent year available is linked through the WRF Users page

http://www.mmm.ucar.edu/wrf/users/download/free_data.html

For additional documentation and information on ordering historical data:

http://dss.ucar.edu/datasets/ds090.0/

b. Running WRFSI

The WRFSI will reliably initialize from the NCEP/NCAR Reanalysis if you carefully follow these instructions.

1. Move the 2D files (*.grb2d.tar -> uncompress to grb2d*) into a separate directory from the 3D files (*.pgb.f##.tar -> pgb.*). WRF will not run without both the 2D surface files and the 3D files. They CANNOT be in the same directory. It is also a good idea to remove any extraneous files from these directories.

2. Run grib_prep.pl (Initial Data step in the SI-GUI) twice – once for the 3D data using the NNRP Vtable and once for the 2D data with the NNRPSFC Vtable. Set this up in the “Sources” tab and make sure the line with the path to your 3D data has 'NNRP' for the 'GRIB source name' and the 'GRIB Vtable used to extract variables'. On the second run, point the path to your 2D files and use 'NNRPSFC'.

3. In the interpolation step, set the interpolation controls for wrfprep.pl in the 'Controls' tab such that

INIT_ROOT = 'NNRP'
LBC_ROOT = 'NNRP'
LSM_ROOT = 'NNRPSFC'
CONSTANTS_FULL_NAME = 

4. Run wrfprep.pl normally

This procedure ensures the surface variables are included in the initialization data that WRF will run off of. If you forget to run grib_prep.pl twice, you will receive skin-temp errors when attempting to run WRF.

**NOTE: SECTION ON SST OVERLAYS AND NAMELIST.INPUT HINTS TEMPORARILY PLACE IN "PART 3 WIKI" DUE TO LENGTH...SEE: http://wikis.lib.ncsu.edu/index.php/Forecasting_Lab_Guide_to_using_the_WRF_Model_at_NC_State_-_Part_3

Acknowledgments

Created by the NCSU Forecasting Lab:

• Dr. Mike Brennan
• Megan Gentry
• Nicole Haglund
• Kevin Hill
• Dr. Gary Lackmann
• Kelly Mahoney

With additional contributions from:

• Matt Borkowski
• Zach Brown
• Dr. Matt Parker