Climate Model Data Services

Several services have been implemented for comparing climate model experiment data.

Set up

• Begin by installing RAMADDA. It can be downloaded here. You can choose several different databases (e.g. MySQL) for the backend. See http://ramadda.org/repository/userguide/database.html for more information.
• You will also need to download, build, and install these additional packages:
  • netCDF C library: http://www.unidata.ucar.edu/downloads/netcdf/index.jsp
  • Climate Data Operators: https://code.zmaw.de/projects/cdo
  • NCAR Command Language (NCL) with OPeNDAP enabled: http://www.ncl.ucar.edu/Download/ (binary release recommended)
  • ImageMagick: http://www.imagemagick.org/script/binary-releases.php

  Be sure to test the installations using the routines supplied with each package.

Configuration

• Disk Space: RAMADDA will use a directory on disk to store the intermediate files and output images from the model comparisons. This will need to hold several days of processed data, so be sure it is large enough (e.g. 500 GB or more). This does not have to be local to the machine, but must be accessible (e.g. NFS mount).
• RAMADDA Resources: You can redirect certain resources from their defaults. (See the RAMADDA User's Guide for more info)

  Disk Resources:

  ramadda.storage.storagedir=/mnt/ramaddaShares/storage
  ramadda.storage.entriesdir=/mnt/ramaddaShares/entries
  ramadda.storage.uploaddir=/mnt/ramaddaShares/uploads
  ramadda.storage.processdir=/mnt/ramaddaShares/process
  

  Program paths: Set the paths to CDO, NCL and ImageMagick's convert

  cdo.path=/usr/local/bin/cdo
  ncl.ncarg_root=/usr/local
  convert.path=/usr/bin/convert
  

Data Harvesting

Once you have set up your RAMADDA server as above, you can start populating the data base with your model output. By convention, the file naming conventions for your climate model output should follow the CMIP5 file naming convention. Each climate model file needs to have metadata associated with it. At a minimum, it needs metadata for:

• Model
• Experiment
• Ensemble member
• Variable

During harvesting, these can be set automatically (see below). Additional fields can be added by defining a new data type.

Each climate model file entry is held in a Climate Model Collection which defines the temporal extent of the data (e.g. Monthly, Daily, 4xDaily). This is set when you create the collection. Currently, the model comparisons only work for monthly output.

How to harvest Climate Model data into RAMADDA

1. Have the data you wish to harvest accessible (e.g. on an NFS mount) on the RAMADDA server.
2. Log into your RAMADDA server with an administrator account.
3. Bring up your RAMADDA server interface in a web browser.
4. Create a new Climate Collection entry to hold the harvested data.
   • Click the dropdown arrow and select the Pick a Type... menu item.
   • Find the Climate Data Collection item in the list, or the specfic type of Climate Collection you want (e.g. NOAA FACTS Climate Data Collection).
   • Give the collection a name, and optionally a description. Also set the time frequency of the collection (e.g. Monthly, Daily)
   • Add the collection.
5. Click on the cog wheel on the page and select the Admin menu item.
6. You must first make the directory where the climate files are stored accessible to the RAMADDA server. From the Admin interface:
   • Select the Settings tab.
   • Expand the Access section on the Settings page.
   • In the File system access box, type in the top level directory or directories where the files reside.
7. From the Admin interface, select the Harvesters tab.
8. Click the "New Harvester" button.
   • Give the new harvester a name.
   • Select the Server File System or a particular type of harvester.
   • Click the Create button to create the harvester.
9. Fill out the fields for the new harvester:
   • Enter the top level directory for the files you want to ingest.
   • Set the file pattern for your files, using a regular expression to indicate the positions for the metadata fields. For example, the NOAA FACTS pattern would look like:
     

     .*/(variable:[1]+)_(model:[2]+)_(experiment:.*)_(ensemble:ens..|mean|sprd|clim)(_([3]+))?.nc

     The C20C+ pattern would look like:

     .*/(variable:[4]+)_([5]+)_(model:[6]+)_(experiment:[7]+)_(subscenario:[8]+)_(subexperiment:[9]+)_(ensemble:[10]+)(_([11]+))?\.nc

   • Set the Base Group to be the Climate Collection that you created above.
   • Set the Entry type to the type of entry you want (e.g. Basic Climate Model File, NOAA FACTS Climate Model File)
   • Do not check the Move file to storage checkbox!
   • Check the Add full metadata checkbox.
   • Set the User to one with admin privileges.
   • Click the Change button to save your options.
10. Click on the Harvesters tab to view the list of harvesters. Your new one should be listed there.
11. Click the Start link to start the harvester. You can reload the page occassionally to view the harvester progress.
12. After the harvester has finished importing the metadata, navigate to the climate collection entry you created. This should show the search interface. If nothing shows in the menus, restart the server and reload the page.

Using the Model Comparison tools

To use the Monthly Climate Model Comparison API, direct your browser to:

http://<your server:port>/repository/model/compare

Your climate collection should be listed under the collections. To go directly to that collection, you can use:

http://<your server:port>/repository/model/compare?collection=<collection id>

where <collection id> is the entry id of the Climate Model Collection.