Unknown Problem Creating +/- 1 Granule Ancillary

Data formats, HDF5, XML profiles, etc.
Post Reply
hcronk
Posts: 19
Joined: Fri Apr 08, 2011 9:03 am

Unknown Problem Creating +/- 1 Granule Ancillary

Post by hcronk »

By looking through the previous exchanges, I am able to produce 1 granule's worth of granulated NCEP and NAAPS data using a portion of the SDR and Masks algorithms. However, I am unable to extend my procedure to create the same data for the surrounding +/- 1 granule, which I need to run the AOT algorithm. I assume the problem is with the VIIRS-MOD-GEO-TC input, as that is the only thing that *should* change. For the 3 granules I am looking at, the other potentially dynamic inputs (NCEP, NAAPS, and Polar Wander files) are the same so the only thing that changes is the geolocation file. I have the 5 consecutive geolocation files and I can reproduce my results consistently for the central granule, but when I switch to the + or - granule, the main PRO_FAIL message in my log says that the required geolocation input is not available:

2012/04/18 16:10:06.753.287 (14742.47781655707360): DBG_HIGH ProCmnMethodAudit.cpp|206|ProCmnAlgorithm[ProAncViirsGranulateTerrainGeopotentialHeight]::handleInputQuery(NPP000129413644) [0x19daca70] PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000129413644 Failure(s): (VIIRS-MOD-RGEO-TC, NPP000129413644), (VIIRS-MOD-GEO-TC, NPP000129413644) from file ProCmnAlgorithm.cpp, line 7699

Obviously I have excessively checked that my 5 geolocation files are consecutive and are not corrupt. Is there something else I can check? I tried scrapping and re-creating the blob files for all 5, separating the groups of 3 into different folders, etc. Not sure what else to try...

Thanks!
Heather
Heather Q. Cronk
NOAA/STAR
kbisanz
Posts: 280
Joined: Wed Jan 05, 2011 7:02 pm
Location: Omaha NE

Re: Unknown Problem Creating +/- 1 Granule Ancillary

Post by kbisanz »

Could you describe your setup a bit? Looks like you're running ProAncViirsGranulateTerrainGeopotentialHeight, but that's part of the VIIRS SDR controller. Are you running the entire VIIRS SDR controller (including Geolocation) or just a piece of it (like the Gran Anc controller)?

In your log file, you'll have a message similar to the below message for each query that's preformed (so there will be a lot). The one to look at will be near the end of the file (since it caused a failure). Some of the text will be different from the below example, but you should look for "DMS query start". In the below example, you can see it's looking for an item with a collection shortname of "VIIRS-CM-IP" (the item 0 line), a granule ID of NPP001212025477 (the item 1 line), and platform shortname and dataset lock values on the last two lines. You should find a query for VIIRS-MOD-RGEO-TC for NPP000129413644. One of the pieces of metadata (on the item N lines) in the query will be mismatched with the geolocation's .asc file on disk. If you're running the entire VIIRS SDR controller (which also creates VIIRS-MOD-RGEO-TC) you probably won't have a .asc file on disk and something else is wrong.

Code: Select all

2011/06/15 20:07:22.767.880 (12479.47052872607184): DBG_LOW DmCoreURMetadataFilter.cpp|140| tid-47052872607184 DMS query start: Searching for items matching the following metadata:
Metadata Collection METADATALIST (size 4)
item 0, 0xc530400 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Collection_Short_Name, comparison=EQ, value=VIIRS-CM-IP
item 1, 0xc5302b8 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP001212025477
item 2, 0xc530170 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=Platform_Short_Name, comparison=EQ, value=NPP
item 3, 0xc5217b0 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=INTEGER name=DatasetLock, comparison=EQ,value=0
You probably don't want to look at the *last* query though. Consider these lines from ProAncViirsGranulateTerrainGeopotentialHeight_CFG.xml

Code: Select all

            <config>
               <name>OfficialShortName_1</name>
               <configValue>VIIRS-MOD-RGEO-TC</configValue>
            </config>
            <config>
               <name>OfficialShortNameConversionProduct_1</name>
               <configValue>VIIRS-MOD-GEO-TC</configValue>
            </config>
First VIIRS-MOD-RGEO-TC (with radian-based angular fields) is queried for. If that is not found, then VIIRS-MOD-GEO-TC (with degree-based angular fields) is searched for. Because it's a "conversion" product, the code knows it must convert the degrees to radians. So, I suspect the last "DMS query start" will be from VIIRS-MOD-GEO-TC and the previous message will be from VIIRS-MOD-RGEO-TC.
Kevin Bisanz
Raytheon Company
houchin
Posts: 128
Joined: Mon Jan 10, 2011 6:20 am

Re: Unknown Problem Creating +/- 1 Granule Ancillary

Post by houchin »

I think it would also be helpful for you to post your execution XML file; are you providing all five granules in the input data but then only creating tasks for the nominal? Or are you trying to make multiple independent ADL runs?
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
hcronk
Posts: 19
Joined: Fri Apr 08, 2011 9:03 am

Re: Unknown Problem Creating +/- 1 Granule Ancillary

Post by hcronk »

Scott,
Here is my execution xml file:

Code: Select all

<InfTkConfig>
  <idpProcessName>ProSdrViirsController.exe</idpProcessName>
  <siSoftwareId></siSoftwareId>
  <isRestart>FALSE</isRestart>
  <useExtSiWriter>FALSE</useExtSiWriter>
  <debugLogLevel>NORMAL</debugLogLevel>
  <dbgDest>D_FILE</dbgDest>
  <debugLevel>DBG_LOW</debugLevel>
  <enablePerf>FALSE</enablePerf>
  <perfPath>${ADL_HOME}/perf</perfPath>
  <dbgPath>${ADL_HOME}/log</dbgPath>
  <initData>
     <domain>OPS</domain>
     <subDomain>SUBDOMAIN</subDomain>
     <startMode>INF_STARTMODE_COLD</startMode>
     <executionMode>INF_EXEMODE_PRIMARY</executionMode>
     <healthTimeoutPeriod>30</healthTimeoutPeriod>
  </initData>
  <lockinMem>FALSE</lockinMem>
  <rootDir>${ADL_HOME}/log</rootDir>
  <inputPath>${ADL_HOME}/data/tiles/Terrain-Eco-ANC-Tile/withMetadata:${ADL_HOME}/data/input/static:${ADL_HOME}/data/input/Camaguey_test_gran/ANC_TEST/minus1</inputPath>
  <outputPath>${ADL_HOME}/data/output/Camaguey_test_gran_SDR/minus1</outputPath>
  <dataStartIET>1422170313000000</dataStartIET>
  <dataEndIET>1422170413000000</dataEndIET>
  <actualScans>48</actualScans>
  <previousActualScans>48</previousActualScans>
  <nextActualScans>48</nextActualScans> 
  <usingMetadata>TRUE</usingMetadata>
  <configGuideName>ProSdrViirs_GuideList.cfg</configGuideName>

  <task>
    <taskType>SDR</taskType>
    <taskDetails1>NPP000129413644</taskDetails1>
    <taskDetails2>A1</taskDetails2>
    <taskDetails3>NPP</taskDetails3>
    <taskDetails4>VIIRS</taskDetails4>
  </task>

  <task>
    <taskType>SHUTDOWN</taskType>
    <taskDetails1></taskDetails1>
    <taskDetails2></taskDetails2>
    <taskDetails3></taskDetails3>
    <taskDetails4></taskDetails4>
  </task>

</InfTkConfig>
Heather Q. Cronk
NOAA/STAR
hcronk
Posts: 19
Joined: Fri Apr 08, 2011 9:03 am

Re: Unknown Problem Creating +/- 1 Granule Ancillary

Post by hcronk »

Kevin,
I am running only the ProAncViirController portion of the VIIRS SDR controller, so I should only need the VIIRS-MOD-GEO-TC files (vs the RGEO files, not only those period) correct?
In my failed attempt logfile, the second to last query is VIIRS-MOD-RGEO-TC and the last is VIIRS-MOD-GEO-TC, as you expected. In the VIIRS-MOD-GEO-TC query, there are a LOT of lines saying that URIDs were excluded because N_Granule_ID was not found. Does it go through and check all my inputs for the Granule ID? Could these messages be from checking the Terrain-Eco-ANC-Tile inputs?

After those messages, there are two lines that read:
2012/04/18 16:10:06.748.463 (14742.47781655707360): DBG_LOW DmCoreURMetadataFilter.cpp|216| tid-47781655707360 Excluding URID 4f8c2a6f-047f8-ca0a14ea-dceb2f71 because the following metadata does not match: 0x1accaf08 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP000129412791
2012/04/18 16:10:06.748.578 (14742.47781655707360): DBG_LOW DmCoreURMetadataFilter.cpp|216| tid-47781655707360 Excluding URID 4f8843a4-84144-ca0a14ea-0cbeee72 because the following metadata does not match: 0x1acd4a50 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP000129414498
I assume this is when it checks my flanking geolocation files and doesn't find the ID it is looking for?

Finally, at the end of the query, it says:
2012/04/18 16:10:06.749.093 (14742.47781655707360): DBG_MED DmCoreURMetadataFilter.cpp|275| tid-47781655707360 DMS query finish: query returned 1 item(s) containing these items:
URID = 4f8843a4-2061b-ca0a14ea-0cb8b349 Collection Short Name = VIIRS-MOD-GEO-TC
2012/04/18 16:10:06.749.160 (14742.47781655707360): DBG_MED ProCmnMethodAudit.cpp|225|ProCmnInputItem[13ProCmnSDRItem]::queryDMS(VIIRS-MOD-GEO-TC) [0x1acea720] PRO_SUCCESS
2012/04/18 16:10:06.749.192 (14742.47781655707360): DBG_MED ProCmnMethodAudit.cpp|123|ProCmnInputItem[13ProCmnSDRItem]::pruneURs(VIIRS-MOD-GEO-TC) [0x1acea720] entered
2012/04/18 16:10:06.749.225 (14742.47781655707360): DBG_MED ProCmnMethodAudit.cpp|173|ProCmnURGranuleFilter[ProCmnURGranuleFilter]::pruneURs() [0x7fff02e70510] entered
2012/04/18 16:10:06.749.291 (14742.47781655707360): DBG_LOW ProCmnURGranuleFilter.cpp|137|ProCmnURGranuleFilter::pruneURs() input: (1 URs)
0 UR:4f8843a4-2061b-ca0a14ea-0cb8b349|CSN:VIIRS-MOD-GEO-TC|GranID:NPP000129413644|Version:A2|Updated:2012-04-13 15:17:56.312889
0 /home/ADL_3.1/ADL/data/input/Camaguey_test_gran/ANC_TEST/minus1/4f8843a4-2061b-ca0a14ea-0cb8b349.VIIRS-MOD-GEO-TC

2012/04/18 16:10:06.749.371 (14742.47781655707360): DBG_LOW ProCmnURGranuleFilter.cpp|183|ProCmnURGranuleFilter::locateGranuleVersion() input: (1 URs)
0 UR:4f8843a4-2061b-ca0a14ea-0cb8b349|CSN:VIIRS-MOD-GEO-TC|GranID:NPP000129413644|Version:A2|Updated:2012-04-13 15:17:56.312889
0 /home/ADL_3.1/ADL/data/input/Camaguey_test_gran/ANC_TEST/minus1/4f8843a4-2061b-ca0a14ea-0cb8b349.VIIRS-MOD-GEO-TC

2012/04/18 16:10:06.749.422 (14742.47781655707360): DBG_LOW ProCmnURGranuleFilter.cpp|466|ProCmnURGranuleFilter::preprocessURCollection() input: (1 URs)
0 UR:4f8843a4-2061b-ca0a14ea-0cb8b349|CSN:VIIRS-MOD-GEO-TC|GranID:NPP000129413644|Version:A2|Updated:2012-04-13 15:17:56.312889
0 /home/ADL_3.1/ADL/data/input/Camaguey_test_gran/ANC_TEST/minus1/4f8843a4-2061b-ca0a14ea-0cb8b349.VIIRS-MOD-GEO-TC
Which looks like it found what it was looking for? I'm having a little trouble deciphering...
Heather Q. Cronk
NOAA/STAR
kbisanz
Posts: 280
Joined: Wed Jan 05, 2011 7:02 pm
Location: Omaha NE

Re: Unknown Problem Creating +/- 1 Granule Ancillary

Post by kbisanz »

I apologize for my delay in posting back.

Yes, if you have only VIIRS-MOD-GEO-TC, the code should convert those and things should work, if everything else is correct.

The querying in ADL 3.1 produces very verbose messages. (The verbosity has been reduced in ADL 4.0) For each piece of metadata in the query (N_Granule_ID, N_Collection_Short_Name, etc) the query looks at the collection of items and starts throwing out items that don't match. When something is thrown out, you get an "Excluded..." message. So yes, you'll get an Excluded message for every input item. While the query looks at the entire set of inputs for the first metadata (say N_Granule_ID), when it comes time to look for the second piece of metadata (say N_Collection_Short_Name) it only looks at the list of inputs that were not excluded by the first pass.

Yes, if you have a bunch of messages saying something didn't have a granule ID, I suspect those are the Terrain-Eco-ANC-Tile inputs.

In this message

Code: Select all

Excluding URID 4f8c2a6f-047f8-ca0a14ea-dceb2f71 because the following metadata does not match: 0x1accaf08 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP000129412791
you could look at 4f8c2a6f-047f8-ca0a14ea-dceb2f71.asc (someplace in your input path) to see the file it's excluding. It's excluding it because the granule ID doesn't match.

The fact that is says

Code: Select all

2012/04/18 16:10:06.749.093 (14742.47781655707360): DBG_MED DmCoreURMetadataFilter.cpp|275| tid-47781655707360 DMS query finish: query returned 1 item(s) containing these items:
means that 1 item made it all the way through the query. Unfortunately, the code still needs to check the "GranuleVersion" metadata. You can see it starting to check the granule version in the lines you posted about "ProCmnURGranuleFilter::pruneURs()" It says that 1 item was input to the pruning process. I suspect if you look farther down it says something about not being able to use that item.

GranuleVersion is a piece of metadata. It has a value such as "A1" or "A2". The first time a granule is processed in the system, it has a granule version of A1. If more data shows up later, that granule may be reprocessed through the system. In that case, the new outputs would have the same granule ID as before (since it is the same granule), but the GranuleVerion would be A2. Normally you'd see an A2 at the end of contact and when the next contact begins. According to someone I asked, "the A2s are happening quite frequently".

In your LW file, on the <taskDetails2> line, you specified A1 (generally the best choice). The problem arises when in the operational system, a granule is reprocessed. So, the geo you got from CLASS I suspect has a GranuleVersion not equal to A1. If you go and look at the .asc files for your geo you should see it.

To fix your issue, you have 2 options:
1. Change your LW file to process A2 granules. The ancillary granulation will probably work ok, but then I believe the granulated data will have a granule version of A2 and you'll have the same set of problems, except with the granulated anc data. :)
2. For the offending Geolocations' .asc files, edit the GranuleVersion to be A1. This of course means you'll slightly deviate from how it's done operationally.

I would recommend option #2. If neither of those options works for you, let me know and maybe we can come up with something.

This poster also had issues with GranuleVersion (the last 2 posts): viewtopic.php?f=32&t=135&p=545#p545

As always, if something doesn't make sense, please post back.
Kevin Bisanz
Raytheon Company
hcronk
Posts: 19
Joined: Fri Apr 08, 2011 9:03 am

Re: Unknown Problem Creating +/- 1 Granule Ancillary

Post by hcronk »

Kevin,
Thanks for that insight! That is not something I ever would have thought to look at! It took care of the problem :)

Thanks!!!
Heather
Heather Q. Cronk
NOAA/STAR
Post Reply