Mx7.1 LUT not imported completely

Issues related to runtime execution of algorithms in ADL
Post Reply
yli
Posts: 16
Joined: Fri Apr 26, 2013 1:25 pm

Mx7.1 LUT not imported completely

Post by yli »

Hi,

In the process of switching from mx6.3 to latest update. I downloaded all the mx7.1 LUTs, extracted and put them all in one directory. I then put this directory in a file to export to INFTK_DM_ROOT just as what I did with mx6.x. For all other files, I keep them unchanged (rdr internal, NCEP, NAAPS, PW_TLE,TILE,etc).

Then I initiate runAdlChainRunnerGui.pl in the ADLmx7.1 directory. One thing I encounter is that most of LUTs are read in, but some are not (in red). For instance, "VIIRS-SDR-DNB-C-COEFFS-LUT" ''VIIRS-SDR-DNB-DN0-LUT" "VIIRS-SDR-DNB-F-PREDICTED-LUT", and a number of others are shown in red for "ProSdrViirsController". Though it seems I can manually import them, this is awkward and I do not understand why some LUTs are identified and some are not. And I attempted to import one of the no-found LUT, a popup window shows "Some metadata for this product has been automatically populated. However more metadata items may be needed. Please review the metadata for correctness."

Has anyone seen this before?
yli
Posts: 16
Joined: Fri Apr 26, 2013 1:25 pm

Re: Mx7.1 LUT not imported completely

Post by yli »

I assume the manual import may work but it turns out not. I ran an actual VIIRS granule but it failed at the first step: ProEdrViirsCloudsFirstController. The error log is posted below, at least related to one of the manually imported LUT
"51b924ae-1f50c-9b9deadd-ed589cb7.VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT_npp_20130304000000Z_20130314000000Z_ee00000000000000Z_PS-1-O-CCR-13-922-JPSS-DPA-006-PE_noaa_all_all-_all.bin".

Can anyone give any comments where I did wrong?

"
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnDataItem[17ProCmnAuxDataItem]::setDataVectorExisting(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] ROOT PRO_FAIL disk file missing and we can't be a shell granule ( your inventory may be corrupted) from file ProCmnDataItem.cpp, line 2677
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnDataItem[17ProCmnAuxDataItem]::mapUR(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] PRO_FAIL setDataVectorExisting() call from file ProCmnDataItem.cpp, line 2461
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnDataItem[17ProCmnAuxDataItem]::mapURs(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] PRO_FAIL mapUR() call from file ProCmnDataItem.cpp, line 1113
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnDataItem[17ProCmnAuxDataItem]::getData(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] PRO_FAIL mapURs() call from file ProCmnDataItem.cpp, line 970
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnInputItem[17ProCmnAuxDataItem]::getData(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] PRO_FAIL ProCmnDataItem::getData() failed from file ProCmnInputItem.cpp, line 194
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnDataItem[17ProCmnAuxDataItem]::get(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] PRO_FAIL getData() call from file ProCmnDataItem.cpp, line 569
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::getDataToConvert(NPP000332351193) [0x2add504bc190] PRO_FAIL no short name found for input: groupname DgAnomalyDnLimitsLut at index 1 from file ProCmnAlgorithm.cpp, line 2195
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::handleInputQuery(NPP000332351193) [0x2add504bc190] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT , Granule ID: Failure(s): (VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT, ) from file ProCmnAlgorithm.cpp, line 7967
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::queryAuxItems(NPP000332351193) [0x2add504bc190] PRO_FAIL call to handleInputQuery() from file ProCmnAlgorithm.cpp, line 8244
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::getDataItems(NPP000332351193) [0x2add504bc190] PRO_FAIL queryAuxItems() call from file ProCmnAlgorithm.cpp, line 1091
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::doIstage(NPP000332351193) [0x2add504bc190] PRO_FAIL getDataItems() call from file ProCmnAlgorithm.cpp, line 3282
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::doIpoModel(NPP000332351193) [0x2add504bc190] PRO_FAIL doIstage() call from file ProCmnAlgorithm.cpp, line 4154
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithm[ProSdrViirsCal]::runAlgorithmImpl() [0x2add504bc190] PRO_FAIL doIpoModel() call from file ProCmnAlgorithm.cpp, line 4274
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithmBase[ProSdrViirsCal]::runAlgorithm() [0x2add504bc190] PRO_FAIL runAlgorithmImpl() call from file ProCmnAlgorithmBase.cpp, line 509
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnControllerAlgorithm[ProSdrViirsController]::runAlgorithmChain(NPP000332351193) [0xc407170] PRO_FAIL runAlgorithm() call from file ProCmnControllerAlgorithm.cpp, line 492
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnControllerBase[ProSdrViirsController]::runAlgorithmImpl(NPP000332351193) [0xc407170] PRO_FAIL runAlgorithmChain() call from file ProCmnControllerBase.cpp, line 243
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAlgorithmBase[ProSdrViirsController]::runAlgorithm(NPP000332351193) [0xc407170] PRO_FAIL runAlgorithmImpl() call from file ProCmnAlgorithmBase.cpp, line 509
Log Message Value ProCmnDataItem[17ProCmnAuxDataItem]::setDataVectorExisting(VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT) [0x2add505cf170] ROOT PRO_FAIL disk file missing and we can't be a shell granule ( your inventory may be corrupted) from file ProCmnDataItem.cpp, line 2677
Log Message Value ProCmnAlgorithm[ProSdrViirsCal]::handleInputQuery(NPP000332351193) [0x2add504bc190] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT , Granule ID: Failure(s): (VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT, ) from file ProCmnAlgorithm.cpp, line 7967
Log Message Value TRACE - (3776.47129653533888): DBG_HIGH ProCmnMethodAudit.cpp|207|ProCmnAppl[ProCmnViirsAppl]::run() [0x7fff4db07990] PRO_FAIL runAlgorithm() call from file ProCmnAppl.cpp, line 774"
bhenders
Posts: 72
Joined: Wed Jan 05, 2011 9:27 am
Location: Omaha, NE

Re: Mx7.1 LUT not imported completely

Post by bhenders »

I will try to help you though this will probably take a few posts to pin point the problem. The granule you are trying to execute is

# The value of NPP_GRANULE_ID_BASETIME affects the granule ID utility which
# If real NPP data is processed by ADL, this variable will need to be
export NPP_GRANULE_ID_BASETIME=1300968033000000
# Use this value for real NPP data.
#export NPP_GRANULE_ID_BASETIME=1698019234000000
~/CdtTalk/ADL/tools/bin > echo $NPP_GRANULE_ID_BASETIME
1698019234000000
~/CdtTalk/ADL/tools/bin > ./granID.exe
Sensor = VIIRS
PSN = VIIRS-SCIENCE
Enter the time or VIIRS Granule ID: NPP000332351193
Input VIIRS Granule ID = NPP000332351193
Output VIIRS Granule ID = NPP000332351193
Start: 1731254353300000 = 2012-11-10 15:58:38.300000Z
End: 1731254438650000 = 2012-11-10 16:00:03.650000Z

However, the 51b924ae-1f50c-9b9deadd-ed589cb7.VIIRS-SDR-DG-ANOMALY-DN-LIMITS-LUT_npp_20130304000000Z_20130314000000Z_ee00000000000000Z_PS-1-O-CCR-13-922-JPSS-DPA-006-PE_noaa_all_all-_all.bin has an effectivity that starts on 20130314000000Z and is open ended for the end effectivity (ee00000000000000Z). This is not the correct version of the LUT needed to execute your 2012 granule. You will need an older LUT version to run the granule you are trying to execute. This is probably why the TK Chain Runner GUI is indicating that the item is "red" and is not available because there is no valid entry in your DMS instance with the correct effectivity. I believe you tried to then import just the binary file and got further but still got the error below. This either eludes to a manual import issue or some other problem. If you want to pursue this further can you turn on debug low and I might be able to tell more on what the issue might be. However, your best path forward is to get the correct versions of the items for the 2012 granule that you are trying to execute.

Respond back if I can help you further.

Thanks,

Bryan Henderson
Raytheon Company
yli
Posts: 16
Joined: Fri Apr 26, 2013 1:25 pm

Re: Mx7.1 LUT not imported completely

Post by yli »

Bryan,

Thank for your instruction. I was not aware of the date requirement to run Mx7.1.

I moved old LUTs, reported missing in Mx7.1, from previous version Mx6.x to where I store Mx7.1 LUT, and now ADL is not complaining about missing LUT. Is this the right way? The test run with an actual granule goes through. I suppose ADL can automatically find right LUTs to use based on date of input data, when multiple same LUTs are in the same location?

Thanks again.

Yue
bhenders
Posts: 72
Joined: Wed Jan 05, 2011 9:27 am
Location: Omaha, NE

Re: Mx7.1 LUT not imported completely

Post by bhenders »

Yue,

The ADL framework will query for Auxillary items (LUT, PCTs, etc) based on the observed date time of the granule being executed. There can be numerous versions of a given LUT in your DMS instance. In your case, you installed the latest LUTs, however, you were still executing a granule that had an observation time in 2012. The most recent version of the LUTs based on their file name effectivity could only be used after March 14, 2013. If you ran a granule after this date, the new version of the LUT would be retrieved.

If someone wants to use the latest lookup table to run data back in 2012, one would have to open up the start effectivity of the LUT to encompass the granule observation time. However, this can be somehwhat dangerous strategy, unless one knows the specific LUT and science details.

The LUT retrieval rules are based on the following:

1. Retrieve all LUTs that have a start/end effectivity time that encompasses the observation time of the granule
2. Use the latest start time effectivity if there is more than one match from above
3. If multiple items with same start effectivity, then break ties based on UpdateDateTime (basically when inserted into DMS), use the most recent item placed into the DMS.

Let me know if you still have further questions.

Thanks,

Bryan Henderson
Raytheon Company
Post Reply