By looking through the previous exchanges, I am able to produce 1 granule's worth of granulated NCEP and NAAPS data using a portion of the SDR and Masks algorithms. However, I am unable to extend my procedure to create the same data for the surrounding +/- 1 granule, which I need to run the AOT algorithm. I assume the problem is with the VIIRS-MOD-GEO-TC input, as that is the only thing that *should* change. For the 3 granules I am looking at, the other potentially dynamic inputs (NCEP, NAAPS, and Polar Wander files) are the same so the only thing that changes is the geolocation file. I have the 5 consecutive geolocation files and I can reproduce my results consistently for the central granule, but when I switch to the + or - granule, the main PRO_FAIL message in my log says that the required geolocation input is not available:
2012/04/18 16:10:06.753.287 (14742.47781655707360): DBG_HIGH ProCmnMethodAudit.cpp|206|ProCmnAlgorithm[ProAncViirsGranulateTerrainGeopotentialHeight]::handleInputQuery(NPP000129413644) [0x19daca70] PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000129413644 Failure(s): (VIIRS-MOD-RGEO-TC, NPP000129413644), (VIIRS-MOD-GEO-TC, NPP000129413644) from file ProCmnAlgorithm.cpp, line 7699
Obviously I have excessively checked that my 5 geolocation files are consecutive and are not corrupt. Is there something else I can check? I tried scrapping and re-creating the blob files for all 5, separating the groups of 3 into different folders, etc. Not sure what else to try...
Thanks!
Heather
Unknown Problem Creating +/- 1 Granule Ancillary
Unknown Problem Creating +/- 1 Granule Ancillary
Heather Q. Cronk
NOAA/STAR
NOAA/STAR
Re: Unknown Problem Creating +/- 1 Granule Ancillary
Could you describe your setup a bit? Looks like you're running ProAncViirsGranulateTerrainGeopotentialHeight, but that's part of the VIIRS SDR controller. Are you running the entire VIIRS SDR controller (including Geolocation) or just a piece of it (like the Gran Anc controller)?
In your log file, you'll have a message similar to the below message for each query that's preformed (so there will be a lot). The one to look at will be near the end of the file (since it caused a failure). Some of the text will be different from the below example, but you should look for "DMS query start". In the below example, you can see it's looking for an item with a collection shortname of "VIIRS-CM-IP" (the item 0 line), a granule ID of NPP001212025477 (the item 1 line), and platform shortname and dataset lock values on the last two lines. You should find a query for VIIRS-MOD-RGEO-TC for NPP000129413644. One of the pieces of metadata (on the item N lines) in the query will be mismatched with the geolocation's .asc file on disk. If you're running the entire VIIRS SDR controller (which also creates VIIRS-MOD-RGEO-TC) you probably won't have a .asc file on disk and something else is wrong.
You probably don't want to look at the *last* query though. Consider these lines from ProAncViirsGranulateTerrainGeopotentialHeight_CFG.xml
First VIIRS-MOD-RGEO-TC (with radian-based angular fields) is queried for. If that is not found, then VIIRS-MOD-GEO-TC (with degree-based angular fields) is searched for. Because it's a "conversion" product, the code knows it must convert the degrees to radians. So, I suspect the last "DMS query start" will be from VIIRS-MOD-GEO-TC and the previous message will be from VIIRS-MOD-RGEO-TC.
In your log file, you'll have a message similar to the below message for each query that's preformed (so there will be a lot). The one to look at will be near the end of the file (since it caused a failure). Some of the text will be different from the below example, but you should look for "DMS query start". In the below example, you can see it's looking for an item with a collection shortname of "VIIRS-CM-IP" (the item 0 line), a granule ID of NPP001212025477 (the item 1 line), and platform shortname and dataset lock values on the last two lines. You should find a query for VIIRS-MOD-RGEO-TC for NPP000129413644. One of the pieces of metadata (on the item N lines) in the query will be mismatched with the geolocation's .asc file on disk. If you're running the entire VIIRS SDR controller (which also creates VIIRS-MOD-RGEO-TC) you probably won't have a .asc file on disk and something else is wrong.
Code: Select all
2011/06/15 20:07:22.767.880 (12479.47052872607184): DBG_LOW DmCoreURMetadataFilter.cpp|140| tid-47052872607184 DMS query start: Searching for items matching the following metadata:
Metadata Collection METADATALIST (size 4)
item 0, 0xc530400 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Collection_Short_Name, comparison=EQ, value=VIIRS-CM-IP
item 1, 0xc5302b8 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP001212025477
item 2, 0xc530170 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=Platform_Short_Name, comparison=EQ, value=NPP
item 3, 0xc5217b0 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=INTEGER name=DatasetLock, comparison=EQ,value=0
Code: Select all
<config>
<name>OfficialShortName_1</name>
<configValue>VIIRS-MOD-RGEO-TC</configValue>
</config>
<config>
<name>OfficialShortNameConversionProduct_1</name>
<configValue>VIIRS-MOD-GEO-TC</configValue>
</config>
Kevin Bisanz
Raytheon Company
Raytheon Company
Re: Unknown Problem Creating +/- 1 Granule Ancillary
I think it would also be helpful for you to post your execution XML file; are you providing all five granules in the input data but then only creating tasks for the nominal? Or are you trying to make multiple independent ADL runs?
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
Re: Unknown Problem Creating +/- 1 Granule Ancillary
Scott,
Here is my execution xml file:
Here is my execution xml file:
Code: Select all
<InfTkConfig>
<idpProcessName>ProSdrViirsController.exe</idpProcessName>
<siSoftwareId></siSoftwareId>
<isRestart>FALSE</isRestart>
<useExtSiWriter>FALSE</useExtSiWriter>
<debugLogLevel>NORMAL</debugLogLevel>
<dbgDest>D_FILE</dbgDest>
<debugLevel>DBG_LOW</debugLevel>
<enablePerf>FALSE</enablePerf>
<perfPath>${ADL_HOME}/perf</perfPath>
<dbgPath>${ADL_HOME}/log</dbgPath>
<initData>
<domain>OPS</domain>
<subDomain>SUBDOMAIN</subDomain>
<startMode>INF_STARTMODE_COLD</startMode>
<executionMode>INF_EXEMODE_PRIMARY</executionMode>
<healthTimeoutPeriod>30</healthTimeoutPeriod>
</initData>
<lockinMem>FALSE</lockinMem>
<rootDir>${ADL_HOME}/log</rootDir>
<inputPath>${ADL_HOME}/data/tiles/Terrain-Eco-ANC-Tile/withMetadata:${ADL_HOME}/data/input/static:${ADL_HOME}/data/input/Camaguey_test_gran/ANC_TEST/minus1</inputPath>
<outputPath>${ADL_HOME}/data/output/Camaguey_test_gran_SDR/minus1</outputPath>
<dataStartIET>1422170313000000</dataStartIET>
<dataEndIET>1422170413000000</dataEndIET>
<actualScans>48</actualScans>
<previousActualScans>48</previousActualScans>
<nextActualScans>48</nextActualScans>
<usingMetadata>TRUE</usingMetadata>
<configGuideName>ProSdrViirs_GuideList.cfg</configGuideName>
<task>
<taskType>SDR</taskType>
<taskDetails1>NPP000129413644</taskDetails1>
<taskDetails2>A1</taskDetails2>
<taskDetails3>NPP</taskDetails3>
<taskDetails4>VIIRS</taskDetails4>
</task>
<task>
<taskType>SHUTDOWN</taskType>
<taskDetails1></taskDetails1>
<taskDetails2></taskDetails2>
<taskDetails3></taskDetails3>
<taskDetails4></taskDetails4>
</task>
</InfTkConfig>
Heather Q. Cronk
NOAA/STAR
NOAA/STAR
Re: Unknown Problem Creating +/- 1 Granule Ancillary
Kevin,
I am running only the ProAncViirController portion of the VIIRS SDR controller, so I should only need the VIIRS-MOD-GEO-TC files (vs the RGEO files, not only those period) correct?
In my failed attempt logfile, the second to last query is VIIRS-MOD-RGEO-TC and the last is VIIRS-MOD-GEO-TC, as you expected. In the VIIRS-MOD-GEO-TC query, there are a LOT of lines saying that URIDs were excluded because N_Granule_ID was not found. Does it go through and check all my inputs for the Granule ID? Could these messages be from checking the Terrain-Eco-ANC-Tile inputs?
After those messages, there are two lines that read:
Finally, at the end of the query, it says:
I am running only the ProAncViirController portion of the VIIRS SDR controller, so I should only need the VIIRS-MOD-GEO-TC files (vs the RGEO files, not only those period) correct?
In my failed attempt logfile, the second to last query is VIIRS-MOD-RGEO-TC and the last is VIIRS-MOD-GEO-TC, as you expected. In the VIIRS-MOD-GEO-TC query, there are a LOT of lines saying that URIDs were excluded because N_Granule_ID was not found. Does it go through and check all my inputs for the Granule ID? Could these messages be from checking the Terrain-Eco-ANC-Tile inputs?
After those messages, there are two lines that read:
I assume this is when it checks my flanking geolocation files and doesn't find the ID it is looking for?2012/04/18 16:10:06.748.463 (14742.47781655707360): DBG_LOW DmCoreURMetadataFilter.cpp|216| tid-47781655707360 Excluding URID 4f8c2a6f-047f8-ca0a14ea-dceb2f71 because the following metadata does not match: 0x1accaf08 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP000129412791
2012/04/18 16:10:06.748.578 (14742.47781655707360): DBG_LOW DmCoreURMetadataFilter.cpp|216| tid-47781655707360 Excluding URID 4f8843a4-84144-ca0a14ea-0cbeee72 because the following metadata does not match: 0x1acd4a50 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP000129414498
Finally, at the end of the query, it says:
Which looks like it found what it was looking for? I'm having a little trouble deciphering...2012/04/18 16:10:06.749.093 (14742.47781655707360): DBG_MED DmCoreURMetadataFilter.cpp|275| tid-47781655707360 DMS query finish: query returned 1 item(s) containing these items:
URID = 4f8843a4-2061b-ca0a14ea-0cb8b349 Collection Short Name = VIIRS-MOD-GEO-TC
2012/04/18 16:10:06.749.160 (14742.47781655707360): DBG_MED ProCmnMethodAudit.cpp|225|ProCmnInputItem[13ProCmnSDRItem]::queryDMS(VIIRS-MOD-GEO-TC) [0x1acea720] PRO_SUCCESS
2012/04/18 16:10:06.749.192 (14742.47781655707360): DBG_MED ProCmnMethodAudit.cpp|123|ProCmnInputItem[13ProCmnSDRItem]::pruneURs(VIIRS-MOD-GEO-TC) [0x1acea720] entered
2012/04/18 16:10:06.749.225 (14742.47781655707360): DBG_MED ProCmnMethodAudit.cpp|173|ProCmnURGranuleFilter[ProCmnURGranuleFilter]::pruneURs() [0x7fff02e70510] entered
2012/04/18 16:10:06.749.291 (14742.47781655707360): DBG_LOW ProCmnURGranuleFilter.cpp|137|ProCmnURGranuleFilter::pruneURs() input: (1 URs)
0 UR:4f8843a4-2061b-ca0a14ea-0cb8b349|CSN:VIIRS-MOD-GEO-TC|GranID:NPP000129413644|Version:A2|Updated:2012-04-13 15:17:56.312889
0 /home/ADL_3.1/ADL/data/input/Camaguey_test_gran/ANC_TEST/minus1/4f8843a4-2061b-ca0a14ea-0cb8b349.VIIRS-MOD-GEO-TC
2012/04/18 16:10:06.749.371 (14742.47781655707360): DBG_LOW ProCmnURGranuleFilter.cpp|183|ProCmnURGranuleFilter::locateGranuleVersion() input: (1 URs)
0 UR:4f8843a4-2061b-ca0a14ea-0cb8b349|CSN:VIIRS-MOD-GEO-TC|GranID:NPP000129413644|Version:A2|Updated:2012-04-13 15:17:56.312889
0 /home/ADL_3.1/ADL/data/input/Camaguey_test_gran/ANC_TEST/minus1/4f8843a4-2061b-ca0a14ea-0cb8b349.VIIRS-MOD-GEO-TC
2012/04/18 16:10:06.749.422 (14742.47781655707360): DBG_LOW ProCmnURGranuleFilter.cpp|466|ProCmnURGranuleFilter::preprocessURCollection() input: (1 URs)
0 UR:4f8843a4-2061b-ca0a14ea-0cb8b349|CSN:VIIRS-MOD-GEO-TC|GranID:NPP000129413644|Version:A2|Updated:2012-04-13 15:17:56.312889
0 /home/ADL_3.1/ADL/data/input/Camaguey_test_gran/ANC_TEST/minus1/4f8843a4-2061b-ca0a14ea-0cb8b349.VIIRS-MOD-GEO-TC
Heather Q. Cronk
NOAA/STAR
NOAA/STAR
Re: Unknown Problem Creating +/- 1 Granule Ancillary
I apologize for my delay in posting back.
Yes, if you have only VIIRS-MOD-GEO-TC, the code should convert those and things should work, if everything else is correct.
The querying in ADL 3.1 produces very verbose messages. (The verbosity has been reduced in ADL 4.0) For each piece of metadata in the query (N_Granule_ID, N_Collection_Short_Name, etc) the query looks at the collection of items and starts throwing out items that don't match. When something is thrown out, you get an "Excluded..." message. So yes, you'll get an Excluded message for every input item. While the query looks at the entire set of inputs for the first metadata (say N_Granule_ID), when it comes time to look for the second piece of metadata (say N_Collection_Short_Name) it only looks at the list of inputs that were not excluded by the first pass.
Yes, if you have a bunch of messages saying something didn't have a granule ID, I suspect those are the Terrain-Eco-ANC-Tile inputs.
In this message
you could look at 4f8c2a6f-047f8-ca0a14ea-dceb2f71.asc (someplace in your input path) to see the file it's excluding. It's excluding it because the granule ID doesn't match.
The fact that is says
means that 1 item made it all the way through the query. Unfortunately, the code still needs to check the "GranuleVersion" metadata. You can see it starting to check the granule version in the lines you posted about "ProCmnURGranuleFilter::pruneURs()" It says that 1 item was input to the pruning process. I suspect if you look farther down it says something about not being able to use that item.
GranuleVersion is a piece of metadata. It has a value such as "A1" or "A2". The first time a granule is processed in the system, it has a granule version of A1. If more data shows up later, that granule may be reprocessed through the system. In that case, the new outputs would have the same granule ID as before (since it is the same granule), but the GranuleVerion would be A2. Normally you'd see an A2 at the end of contact and when the next contact begins. According to someone I asked, "the A2s are happening quite frequently".
In your LW file, on the <taskDetails2> line, you specified A1 (generally the best choice). The problem arises when in the operational system, a granule is reprocessed. So, the geo you got from CLASS I suspect has a GranuleVersion not equal to A1. If you go and look at the .asc files for your geo you should see it.
To fix your issue, you have 2 options:
1. Change your LW file to process A2 granules. The ancillary granulation will probably work ok, but then I believe the granulated data will have a granule version of A2 and you'll have the same set of problems, except with the granulated anc data.
2. For the offending Geolocations' .asc files, edit the GranuleVersion to be A1. This of course means you'll slightly deviate from how it's done operationally.
I would recommend option #2. If neither of those options works for you, let me know and maybe we can come up with something.
This poster also had issues with GranuleVersion (the last 2 posts): viewtopic.php?f=32&t=135&p=545#p545
As always, if something doesn't make sense, please post back.
Yes, if you have only VIIRS-MOD-GEO-TC, the code should convert those and things should work, if everything else is correct.
The querying in ADL 3.1 produces very verbose messages. (The verbosity has been reduced in ADL 4.0) For each piece of metadata in the query (N_Granule_ID, N_Collection_Short_Name, etc) the query looks at the collection of items and starts throwing out items that don't match. When something is thrown out, you get an "Excluded..." message. So yes, you'll get an Excluded message for every input item. While the query looks at the entire set of inputs for the first metadata (say N_Granule_ID), when it comes time to look for the second piece of metadata (say N_Collection_Short_Name) it only looks at the list of inputs that were not excluded by the first pass.
Yes, if you have a bunch of messages saying something didn't have a granule ID, I suspect those are the Terrain-Eco-ANC-Tile inputs.
In this message
Code: Select all
Excluding URID 4f8c2a6f-047f8-ca0a14ea-dceb2f71 because the following metadata does not match: 0x1accaf08 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Granule_ID, comparison=EQ, value=NPP000129412791
The fact that is says
Code: Select all
2012/04/18 16:10:06.749.093 (14742.47781655707360): DBG_MED DmCoreURMetadataFilter.cpp|275| tid-47781655707360 DMS query finish: query returned 1 item(s) containing these items:
GranuleVersion is a piece of metadata. It has a value such as "A1" or "A2". The first time a granule is processed in the system, it has a granule version of A1. If more data shows up later, that granule may be reprocessed through the system. In that case, the new outputs would have the same granule ID as before (since it is the same granule), but the GranuleVerion would be A2. Normally you'd see an A2 at the end of contact and when the next contact begins. According to someone I asked, "the A2s are happening quite frequently".
In your LW file, on the <taskDetails2> line, you specified A1 (generally the best choice). The problem arises when in the operational system, a granule is reprocessed. So, the geo you got from CLASS I suspect has a GranuleVersion not equal to A1. If you go and look at the .asc files for your geo you should see it.
To fix your issue, you have 2 options:
1. Change your LW file to process A2 granules. The ancillary granulation will probably work ok, but then I believe the granulated data will have a granule version of A2 and you'll have the same set of problems, except with the granulated anc data.
2. For the offending Geolocations' .asc files, edit the GranuleVersion to be A1. This of course means you'll slightly deviate from how it's done operationally.
I would recommend option #2. If neither of those options works for you, let me know and maybe we can come up with something.
This poster also had issues with GranuleVersion (the last 2 posts): viewtopic.php?f=32&t=135&p=545#p545
As always, if something doesn't make sense, please post back.
Kevin Bisanz
Raytheon Company
Raytheon Company
Re: Unknown Problem Creating +/- 1 Granule Ancillary
Kevin,
Thanks for that insight! That is not something I ever would have thought to look at! It took care of the problem
Thanks!!!
Heather
Thanks for that insight! That is not something I ever would have thought to look at! It took care of the problem
Thanks!!!
Heather
Heather Q. Cronk
NOAA/STAR
NOAA/STAR