ADL 4.0 Ingest NCEP grib data

[rnc]

Hi there -

I have run into difficulty ingesting NCEP data from grib2 files and I hope someone can help me. I already have this workign sort-of but am trying to make my pipeline more efficient. The process that works for me is to obtain h5 wrapped NCEP and use the perl script DynAncConverter.pl to convert the format. However, obtaining the HDF5 seems to require operator intensive interaction with CLASS. I am trying to see if I can use the grib formatted data from http://jpssdb.ssec.wisc.edu/ancillary/ which can be downloaded from a script-able wget command.

So my first question is whether the grib files have all the information I need to seed the computation of a CrIMSS EDR. I wonder if they do for the following reason: When I use the python script NCEPtoBlob.py I get an error that the adl_struct has no geopotentialHeightLayers attribute.

Assuming I'm not attempting the impossible, can someone point me to the right combination of scripts to convert the grib data? Based on https://jpss-adl-wiki.ssec.wisc.edu/med ... _Home#NCEP I have tried DynAncConverter.pl on the file and it seemed to work -- but did not build the output in the format ADL needed to compute the EDR.

Many thanks

Rich Czerwinski
MIT Lincoln Laboratory

[tjensen]

Hey Rich.

I'm not familiar with the scripts you mentioned using (DynAncConverter.pl and NCEPtoBlob.py) or the exact contents of the grib data that is hosted by UW. Looking at what's there and conferring with some others here at Raytheon, I think the data there should work if properly Ingested. I'd suggest trying to run the data through the runMsd.pl script to convert it and see if that fixes your problem.

If you need a refresher on using the runMsd.pl script, see the following posts for more info:
viewtopic.php?f=32&t=266&p=925&hilit=runmsd#p925
viewtopic.php?f=27&t=252&p=926&hilit=runmsd#p926

Hope that helps!

[geoffc]

Hi Rich,
I am the developer of NCEPtoBlob.py. Could you provide a link to one of the grib files in http://jpssdb.ssec.wisc.edu/ancillary/ which resulted in the "adl_struct has no geopotentialHeightLayers attribute" error.
Also, you can the latest versions of NCEPtoBlob.py (and the required adl_blob.py) from...

https://svn.ssec.wisc.edu/repos/jpss_ad ... PtoBlob.py
https://svn.ssec.wisc.edu/repos/jpss_ad ... dl_blob.py

Thanks,
Geoff

[rnc]

Hi Tim, Goeff
Thanks for your responses.

Geoff: I find I am now unable to replicate the NCEPtoBlob.py error, so while I appreciate the offer, I (un)fortunately need to withdraw the request. I'll get in touch if that particular problem resurfaces.

Tim: I ran across the MSD instructions when you and Kevin were helping me with my CrIS problem, and I did think of that. I'm a little confused how to jump into the conversation you pointed me to... it seems to be related to processing an .h5 file to MSD and then making use of runMSD. In contrast, I have a .grib file, and when I use the tools DynAncConverter.pl and NCEPtoBlob.py (recommended per http://jpss.ssec.wisc.edu/adl/download/ ... toBLOB.pdf) the data pops out in ADL binary / ascii format. That process does seem to work -- but ProEdrCrimssArtmProgController throws an error which looks like either a failed or wrong NCEP ingest.

I'll attach the log file... maybe you can just look at that and see that I'm grabbing the wrong NCEP file or something.

Thanks again

Rich

[tjensen]

The runMsd.pl script should handle .grib files. You should just need to put the .grib files in the ADL Landing Zone (the ANC directory I believe) and it should do the necessary conversions.

Looking at your log, it seems to be unable to find the matching NCEP file that it needs:

DMS query start: Searching for items matching the following metadata:
Metadata Collection METADATALIST (size 3)
item 0, 0x7b29df0 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Collection_Short_Name, comparison=EQ, value=NCEP-ANC-Int
item 1, 0x81a28a0 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=INTEGER name=DatasetLock, comparison=EQ,value=0
item 2, 0x80e3410 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=DATETIMERANGE name=Effectivity, comparison=EQ2012-10-01 03:02:19.313000,2012-10-01 03:02:19.313000

2012/10/18 18:45:06.735.168 (11949.47734340508320): DBG_MED DmCoreURMetadataFilter.cpp|258| tid-47734340508320 DMS query finish: query returned 0 item(s) containing these items:
2012/10/18 18:45:06.735.329 (11949.47734340508320): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnInputItem[16ProCmnAncIntItem]::queryDMS(NCEP-ANC-Int) [0x817b9a0] UNKNOWN_ERROR queryMetadata: [N_Collection_Short_Name:EQ:NCEP-ANC-Int] AND [DatasetLock:EQ:0] AND [Effectivity:EQ:[2012-10-01 03:02:19.313000 - 2012-10-01 03:02:19.313000]] from file ProCmnInputItem.cpp, line 430

I recommend double checking the metadata in the .asc file you think should have the right Collection Short Name, DatasetLock and Effectivity. Note that with Effectivity, your NCEP file likely won't exactly match the date & time that is being queried for. The NCEP file's effectivity range just needs to include the time that is queried for.

If you want help looking at the metadata, just let me know.

[rnc]

Thanks Tim,
The effectivity range is definitely my problem. I processed a granule within that interval and the EDR now runs to completion. However I'm confused now about a different point -- none of the NCEP forecasts in http://jpssdb.ssec.wisc.edu/ancillary/*/gfs*.grib2 have effectivity range greater than an hour. What is the process for interpolating a file that will work for a granule that does not line up with an available NCEP?

Rich

[rnc]

OK... I'm in the process of answering my own question. I recalled I saw something called DynAncInterpolator.pl which seems like it should do the trick. It seems like it's missing something though that I need to install.... Stay tuned, please -- and thanks again for all the help.

Rich

[rnc]

Hi again -
I thought I had completely solved this... I've got a script running now that processes nearly 350 different granules all the way through to CrIMSS EDR. They are all from Jan-Mar 2012 so no CrIS data is included, but I have also made CrIMSS with CrIS present as well, so I have some faith in that process as well.

Unfortunately, the ProEdrCrimssAtmProfController program does not run on a small number of the granules I am interested in. The error is the same as I was encountering before: the Effectivity range on the NCEP file is invalid. I'm attaching a log file for reference, but my question is maybe more succinct: what time is it that gets compared with the NCEP effectivity? I am using DynAncInterpolator.pl to interpolate the NCEP files to an interval that spans the time stamp in the granule filename. However, the affected granules all seem to be near a seam in the interpolation -- so I'm wondering if there's a different time stamp somewhere that does not match the one in the filename.

In a possibly related question, I noticed something funny about the interpolated NCEP file Effectivities. I am using DynAncInterpolator.pl to process two 6 hour predictions spanning the granule time. The NCEP files are time stamped every three hours, so the interpolation generates four NCEP-Int files. THe following are the Effectivities:

5093f93b-38c91-229b0fa9-8c2d70d5.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 08:30:00.000000" "2012-01-03 09:29:59.999999")
5093fa8e-8e77e-229b0fa9-33cde765.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 09:45:00.000000" "2012-01-03 10:29:59.999999")
5093f93b-41ecb-229b0fa9-63b4069f.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 11:30:00.000000" "2012-01-03 12:29:59.999999")
5093fa8e-a8de3-229b0fa9-33cf8dca.asc: ("Effectivity" DATETIMERANGE EQ "2012-01-03 10:30:00.000000" "2012-01-03 11:29:59.999999")

Why does the second file start at 09:45 not 09:30, and is this related to the present problem? The time stamp on the granule *is* right around 09:30...

Thanks again for all the help

Rich

[kbisanz]

Do all the granules where you're having issues occur between 9:30 and 9:45? It appears you don't have coverage of NCEP data for that time range. The one file starting at 9:45 (vs 9:30) is intentional. The temporal interpolation software is designed to work that way. It's pretty complicated and you're not the first person to have this question. The basic problem is that NCEP can't be used until it is 3 hours and 45 minutes old (according to some rule in the system). So you can only use the 3HR forcast after it's 3:45 old. So instead of the 3HR forecast starting at 9:30, it's not allowed to start until 9:45.

I believe you have two options to fix your issue. You only need to perform one of them:
1) Edit $ADL_HOME/cfg/ANC_CFG_Shared.xml. Find TIME_UNTIL_USE for NCEP data and change the value to 0. Note that this is NOT what the operational code does, so your results will be different from what is produced in operations.
2) Get more NCEP from an earlier model run, ingest it, and temporally interpolate it. This is what the operational code does. You should be able to look at the metadata on the CrIMSS EDR from CLASS and see what NCEP files it used.

Here is the answer to a similar question we received via email involving NAAPS data. Note that in the below email, a certain time length is 4:15 for NAAPS, but it'll be 3:45 for you with NCEP data. I'm not sure if it will help explain things or confuse things more.

Paul,

I don't think there is an issue. Basically I think Sid just needs more NAAPS files get the effectivity coverage he desires.

Based on the native NAAPS files Sid provided:

off_NAAPS-03HR-ANC_NAAPS_FNMOC_003f_20120811_201208110600Z_20120811091500Z_ee20120811120000Z_np.h5
off_NAAPS-03HR-ANC_NAAPS_FNMOC_003f_20120811_201208111200Z_20120811151500Z_ee20120811180000Z_np.h5
off_NAAPS-06HR-ANC_NAAPS_FNMOC_006f_20120811_201208110600Z_20120811091500Z_ee20120811150000Z_np.h5
off_NAAPS-06HR-ANC_NAAPS_FNMOC_006f_20120811_201208111200Z_20120811151500Z_ee20120811210000Z_np.h5
off_NAAPS-09HR-ANC_NAAPS_FNMOC_009f_20120811_201208110000Z_20120811031500Z_ee20120811120000Z_np.h5
off_NAAPS-09HR-ANC_NAAPS_FNMOC_009f_20120811_201208110600Z_20120811091500Z_ee20120811180000Z_np.h5
off_NAAPS-09HR-ANC_NAAPS_FNMOC_009f_20120811_201208111200Z_20120811151500Z_ee20120812000000Z_np.h5

The 9 hour 00Z file will cover a 08:30 - 09:30 effectivity.
The 3, 6 and 9 hours 06Z files will cover a 10:15 - 15:30 effectivity.
The 3, 6, and 9 hour 12Z files will cover a 16:15 - 21:30 effectivity.

To get effectivity coverage from 21:30 - 08:30, either additional future forecast files for the above synoptic times are needed, or additional 18Z-based forecast files are needed.

The gaps between 9:30 and 10:15, and 15:30 and 16:15 are due to ING MSD having a configured 4:15 "minimum time until use" for NAAPS data. The solution to cover these gaps would be to also have the 12 hour files for 00Z and 06Z.

The 4:15 usage limit comes from the EDR-IR, the EDR-IR specifies a minimum time that must elapse before certain ancillary data is to be used after being produced. This manifests itself in IDPS through a manipulation of the start effectivity of certain ancillary products with a given forecast. For NCEP data, this time value is 3:45.

Both the time-until-use and effectivity window values are configurable items in the $ADL_HOME/cfg/ANC_CFG_Shared.xml. They are currently set to the operational values.
TIME_UNTIL_USE
INTERNAL_FORECAST_WINDOW

I'll try to describe the scenario we are looking at in more detail below:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
There are 3, 6 and 9 hour native NAAPS files for 06Z. The expected interpolated hourly forecasts would be 4, 5, 7, and 8, which were created based on seeing the following metadata .asc output files. Also, they appear have the expected effectivity times.

-- 3 hr, not created due to the NAAPS 4 hour and 15 minute usage limit
*6fd.asc -- 4 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 10:15:00.000000" "2012-08-11 10:29:59.999999")
*c8a.asc -- 5 hr,("Effectivity" DATETIMERANGE EQ "2012-08-11 10:30:00.000000" "2012-08-11 11:29:59.999999")
-- 6 hr, was not in Sid's output, but I assume it was created since it's needed to create the 4 and 5 hour forecasts. Effectivity would be 11:30 - 12:30
*89e.asc -- 7 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 12:30:00.000000" "2012-08-11 13:29:59.999999")
*dcf.asc -- 8 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 13:30:00.000000" "2012-08-11 14:29:59.999999")
-- 9 hr, was not in Sid's output, but I assume it was created since it's needed to create the 7 and 8 hour forecasts. Effectivity would be 14:30 - 15:30

Note that for NAAPS data, there is a 4 hour and 15 minute usage limit, so that is why the effectivity of the 4 hour forecast above starts at 10:15 (6:00 + 4:15 = 10:15), and why there is no 3 hour forecast. The rest of the interpolated forecasts all have 1 hour effectivity ranges, as expected.

There are 3, 6 and 9 hour NAAPS files for 12Z. The expected interpolated hourly forecasts would be 4, 5, 7, and 8, which were created based on seeing the following metadata .asc output files. Also, they have the expected effectivity times.

-- 3 hr, not created due to the NAAPS 4 hour and 15 minute usage limit
*1af.asc -- 4 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 16:15:00.000000" "2012-08-11 16:29:59.999999")
*aae.asc -- 5 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 16:30:00.000000" "2012-08-11 17:29:59.999999")
-- 6 hr, was not in Sid's output, but I assume it was created since it's needed to create the 4 and 5 hour forecasts. Effectivity would be 17:30 - 18:30
*cd8.asc -- 7 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 18:30:00.000000" "2012-08-11 19:29:59.999999")
*948.asc -- 8 hr, ("Effectivity" DATETIMERANGE EQ "2012-08-11 19:30:00.000000" "2012-08-11 20:29:59.999999")
-- 9 hr, was not in Sid's output, but I assume it was created since it's needed to create the 7 and 8 hour forecasts. Effectivity would be 20:30 - 21:30

Note that for NAAPS data, there is a 4 hour and 15 minute usage limit, so that is why the effectivity of the 4 hour forecast above starts at 16:15 (12:00 + 4:15 = 16:15), and why there is no 3 hour forecast. The rest of the interpolated forecasts all have 1 hour effectivity ranges, as expected.

There is a 9 hour forecast NAAPS file for 00Z. There won't be any interpolated forecasts because there is no other 00Z files. So we would get a single 00Z 9 hour forecast . The effectivity would be 08:30 - 09:30

So in the end, there is a total effectivity range from 08:30 to 09:30, 10:15 to 15:30, 16:15 to 21:30

The gaps between 9:30 and 10:15, and 15:30 and 16:15 are due to the NAAPS 4:15 usage limit. Again, I believe the solution to cover these gaps would be to also have the 12 hour files for 00Z and 06Z.

Also, if there were 3 and 6 hour 00Z files, then we would have additional effectivity coverage from 04:15 - 08:30 .

[rnc]

Hi Kevin
The problematic granules are not exclusively in that excluded interval, however most of them are close enough (08:57 - 09:41, 11:28) that it made me wonder if there was a hidden time stamp that was inconsistent with the file name. Note these are not all the same day - they come from multiple days in Jan 2012, and of the 350ish granules I did run successfully, none of them had a time stamp between 0857 and 0941 (the 1128 might have failed for a different reason; I'm going to come back to that one...)

The one in the example I sent has ("RangeDateTime" DATETIMERANGE EQ "2012-01-03 09:29:25.542000" "2012-01-03 09:29:57.539000"). I set the TIME_UNTIL_USE to 0 as you suggested (because it seemed a quicker way to check your suggestion) and I still get an error from ProEdrCrimssAtmProfController.exe.

I appreciate all your help on this

Rich