run Clouds Mask using SDRs as its input

[bhenders]

I looked at your log file and the problem is still due to the GridToGran Snow output being a shell item. It's hard to tell from the debug high output exactly what is going on but I have a new theory. Especially, since you've indicated that other granules are now working.

The failure is still occurring, I believe, because when you ran previously with the M1 day only band you produced some shell outputs for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran. The automation framework that we put in place checks for the existence of the GridToGran products and Granulate Ancillary items before creating them. If they can be found in the system, it will not create new ones but rather just re-use the available ones. I suspect that these granules continue to fail because you are re-using the shell items that were produced previously in error.

In order to correct it you can just remove all of the VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran products that you've produced previously that have Shell_Status metadata set to "Yes" to ensure all of the bad shell ones get cleaned up.

You should be able to execute the following:

$ADL_HOME/script/DMS/RemoveFromDms.plx -csn VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran -ss Yes

Which should remove all of the items for that product that are shell in your DMS. That should allow this scenario to work for you for all granules.

Thanks,

Bryan Henderson
Raytheon Company

[wzchen]

OK. I tried this tool. However, it didn't work. If I enable "perl_env" in envSetup.ksh file, it complained that some of the ADL's perl modules have problems.

Bareword found where operator expected at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8_heavy.pl line 21, near "s/[-\s_]//rg"
syntax error at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8_heavy.pl line 21, near "s/[-\s_]//rg"
BEGIN not safe after errors--compilation aborted at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8_heavy.pl line 147.
Compilation failed in require at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8.pm line 17.
BEGIN failed--compilation aborted at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/constant.pm line 45.
Compilation failed in require at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/Getopt/Long.pm line 208.
BEGIN failed--compilation aborted at /data/data020/weizhong/ADL4.1/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/Getopt/Long.pm line 208.
Compilation failed in require at /data/data020/weizhong/ADL4.1/CSPP/ADL4.1_Mx6.7_VCM_sdr_1/script/DMS/RemoveFromDms.plx line 86.
BEGIN failed--compilation aborted at /data/data020/weizhong/ADL4.1/CSPP/ADL4.1_Mx6.7_VCM_sdr_1/script/DMS/RemoveFromDms.plx line 86.

However, I never change utf8_heavy.pl file. Here is its line 21:

Code: Select all

my $loose = $_[0] =~ s/[-\s_]//rg;

If I disable "perl_env" in envSetup.ksh file, it complained:

Error: Cannot find the DMS root directory:

Which variable is set for DMS root? I only find "DPE_ROOT_PATH" in envSetup.ksh.

Thanks.

[tjensen]

The DMS root directory is based on the INFTK_DM_ROOT environment variable (the same one used by the Chain Runner to find your inputs). I believe the DMS scripts will throw that error if one of the directories in the INFTK_DM_ROOT does not exist. Check what your INFTK_DM_ROOT is set to and make sure it only contains valid directories.

I'm not sure about the Perl problem. The only Perl related environment variable in the envSetup.ksh delivered by Ratheon "PERL_HOME". Looking at the paths in your errors, I believe CSPP is a UW tool built on top of ADL. Perhaps this is something that was added by UW. However, if you're getting to the DMS root directory error without this enabled, I think the script should still work without it.

[wzchen]

I changed some source code under the "cloudMask/src" directory and its corresponding code under the "cloudMask/include" directory. I am wondering if I can just rebuild cloudMask algorithm without rebuild the whole ADL, like you mentioned before:

cd $ADL_HOME/EDR/VIIRS/imagery/cloudMask/src

make clean src library

Thanks.

[kbisanz]

It kind of depends what exactly you changed. If you changed .h files that are only included by .cpp files in $ADL_HOME/EDR/VIIRS/imagery/cloudMask/src, then yes you only need to rebuild in $ADL_HOME/EDR/VIIRS/imagery/cloudMask/src. However, if you changed a .h file that is included by something outside of cloud mask, you would need to rebuild that also. Be aware of the possibility that a file might be indirectly included.

You could try rebuilding only $ADL_HOME/EDR/VIIRS/imagery/cloudMask/src, but if you start having weird runtime issues, I would recommend a full ADL rebuild. To be safe, a full rebuild when you went for lunch or left for the day would be safest.

[geoffc]

In another post...

viewtopic.php?f=31&t=387&p=1404#p1401

I raised an issue with ADL not respecting the config value controlling the type of distance calculation is the common adjacency code. I changed the relevant value in PRO_CMN_ADJ_CFG.xml from "GROWCOL" to "GEO". As per previous posts in this thread I removed the "GridRowCol_Mod" entries in ProEdrViirsActiveFires_CFG.xml and ProEdrViirsCM_CFG.xml, and recompiled the ProEdrViirsMasksController thus...

Code: Select all

cd ${ADL_HOME}/EDR/VIIRS/land/ActiveFires/src
make -f Makefile clean library
cd ${ADL_HOME}/EDR/VIIRS/imagery/cloudMask/src
make -f Makefile clean library

cd ${ADL_HOME}/EDR/VIIRS/imagery/Controller/src
make -f Makefile clean program

I ran ProEdrViirsMasksController.exe, and am still getting core dumps, with the log message...

Code: Select all

014/02/04 19:24:14.881.765 (27715.47449750720480): DBG_MED ProCmnAdjFactory.cpp|223|Distance calculation type = GROWCOL
2014/02/04 19:24:14.882.799 (27715.47449750720480): DBG_MED ProCmnAdjFactory.cpp|261|Number of cross granule scans = 2
2014/02/04 19:24:14.883.298 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|124|ProCmnAlgorithm[ProEdrViirsActiveFires]::getAnInputItemFromMe(NPP000562835454) [0x24a2000] entered
2014/02/04 19:24:14.883.328 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|219|ProCmnAlgorithm[ProEdrViirsActiveFires]::getAnInputItemFromMe(NPP000562835454) [0x24a2000] PRO_SUCCESS
2014/02/04 19:24:14.883.381 (27715.47449750720480): DBG_LOW ProCmnDataItem.cpp|1496|ProCmnDataItem::dataPtrEstablished(VIIRS-MOD-GRC-TC) 0x24af540 index 0 has data pointer established
2014/02/04 19:24:14.883.419 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|124|ProCmnAlgorithm[ProEdrViirsActiveFires]::getAnInputItemFromMe(NPP000562835454) [0x24a2000] entered
2014/02/04 19:24:14.883.453 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|219|ProCmnAlgorithm[ProEdrViirsActiveFires]::getAnInputItemFromMe(NPP000562835454) [0x24a2000] PRO_SUCCESS
2014/02/04 19:24:14.883.469 (27715.47449750720480): DBG_LOW ProCmnDataItem.cpp|1496|ProCmnDataItem::dataPtrEstablished(VIIRS-MOD-GRC-TC) 0x24af770 index 0 has data pointer established
2014/02/04 19:24:14.883.571 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|124|ProCmnAlgorithm[ProEdrViirsActiveFires]::getAnInputItemFromMe(NPP000562835454) [0x24a2000] entered
2014/02/04 19:24:14.883.587 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|219|ProCmnAlgorithm[ProEdrViirsActiveFires]::getAnInputItemFromMe(NPP000562835454) [0x24a2000] PRO_SUCCESS
2014/02/04 19:24:14.883.602 (27715.47449750720480): DBG_LOW ProCmnDataItem.cpp|1496|ProCmnDataItem::dataPtrEstablished(VIIRS-MOD-GRC-TC) 0x24afdf0 index 0 has data pointer established
2014/02/04 19:24:14.885.601 (27715.47449750720480): DBG_LOW ProCmnAdjMODTable.cpp|142|cmn adj distance search type = GROWCOL
2014/02/04 19:24:14.895.374 (27715.47449750720480): DBG_LOW ProCmnAdjMODTable.cpp|215| grid_to_latlon : grid_to_latlon, 200: grid_type=0, is illegal
...
2014/02/04 19:24:16.689.061 (27715.47449750720480): DBG_HIGH ProCmnAdjTable.cpp|163|Trying to set to an invalid pixel, 261,2575
...

and

Code: Select all

2014/02/04 19:25:35.845.577 (27715.47449750720480): DBG_MED Cloud_Adjacency.cpp|141|Cloud_Adjacency -- Entered
2014/02/04 19:25:35.848.488 (27715.47449750720480): DBG_MED ProCmnAdjFactory.cpp|223|Distance calculation type = GROWCOL
2014/02/04 19:25:35.848.871 (27715.47449750720480): DBG_MED ProCmnAdjFactory.cpp|261|Number of cross granule scans = 5
2014/02/04 19:25:35.848.956 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|124|ProCmnAlgorithm[ProEdrViirsCM]::getAnInputItemFromMe(NPP000562835454) [0x2b27dc000010] entered
2014/02/04 19:25:35.848.988 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|219|ProCmnAlgorithm[ProEdrViirsCM]::getAnInputItemFromMe(NPP000562835454) [0x2b27dc000010] PRO_SUCCESS
2014/02/04 19:25:35.849.016 (27715.47449750720480): DBG_LOW ProCmnDataItem.cpp|1496|ProCmnDataItem::dataPtrEstablished(VIIRS-MOD-GRC) 0x27e8a40 index 0 has data pointer established
2014/02/04 19:25:35.849.094 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|124|ProCmnAlgorithm[ProEdrViirsCM]::getAnInputItemFromMe(NPP000562835454) [0x2b27dc000010] entered
2014/02/04 19:25:35.849.112 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|219|ProCmnAlgorithm[ProEdrViirsCM]::getAnInputItemFromMe(NPP000562835454) [0x2b27dc000010] PRO_SUCCESS
2014/02/04 19:25:35.849.127 (27715.47449750720480): DBG_LOW ProCmnDataItem.cpp|1496|ProCmnDataItem::dataPtrEstablished(VIIRS-MOD-GRC) 0x2701440 index 0 has data pointer established
2014/02/04 19:25:35.849.142 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|124|ProCmnAlgorithm[ProEdrViirsCM]::getAnInputItemFromMe(NPP000562835454) [0x2b27dc000010] entered
2014/02/04 19:25:35.849.157 (27715.47449750720480): DBG_MED ProCmnMethodAudit.cpp|219|ProCmnAlgorithm[ProEdrViirsCM]::getAnInputItemFromMe(NPP000562835454) [0x2b27dc000010] PRO_SUCCESS
2014/02/04 19:25:35.849.172 (27715.47449750720480): DBG_LOW ProCmnDataItem.cpp|1496|ProCmnDataItem::dataPtrEstablished(VIIRS-MOD-GRC) 0x2708640 index 0 has data pointer established
2014/02/04 19:25:35.849.653 (27715.47449750720480): DBG_LOW ProCmnAdjMODTable.cpp|142|cmn adj distance search type = GROWCOL

after which I get a core dump. The above log entries are the same as before I made the changes to ProEdrViirsActiveFires_CFG.xml and ProEdrViirsCM_CFG.xml

Do I need to do a full rebuild of ADL for the common adjacency change to have effect, or am I not rebuilding the Masks controller properly?

Cheers,
Geoff

[kbisanz]

You appear to be doing things correctly. Can you verify that $DCONFIG points to the location of your config guides?

If you edited the Active Fires and Cloud Mask config guides contained in $DCONFIG to not list a GRC input, then we shouldn't be seeing log messages containing "dataPtrEstablished(VIIRS-MOD-GRC-TC)". After the config change you would need to rebuild the algorithms. It appears you rebuilt the correct locations.

Please check if DCONFIG is correct and post back.

FYI: A PCR has been opened for the original scenario not working correctly. It is PCR037418.

[geoffc]

The value of DCONFIG I was using was ok, but as we have a custom cfg file location, I was editing the usual ADL cfg location and not our new custom location. I made the changes to ProEdrViirsActiveFires_CFG.xml, ProEdrViirsCM_CFG.xml and PRO_CMN_ADJ_CFG.xml, and recompiled, and the Masks controller now completes successfully.

This thread leads me to ask another question: previously I was just editing PRO_CMN_ADJ_CFG.xml to substitute "GEO" for "GROWCOL" for the distance calculation type. Would this result in the the GRC files still being required as a dependency, just not used at runtime? I would make dummy VIIRS-MOD-GRC files, which allowed the Masks controller to run, and assumed the bad GRC values weren't being used due to the "GEO" setting in PRO_CMN_ADJ_CFG.xml. Does removing the VIIRS-MOD-GRC related entries in ProEdrViirsActiveFires_CFG.xml and ProEdrViirsCM_CFG.xml mean that VIIRS-MOD-GRC files are no longer needed? Our VIIRS GTM developer still needs to run the VIIRS SDR to get the GRC files required for that controller, it would be nice if we could switch off the GRC files as a dep for that too.

Cheers, Geoff

[kbisanz]

The reason you needed to remove the GRC entries from ProEdrViirsActiveFires_CFG.xml and ProEdrViirsCM_CFG.xml and recompile is that at build time those config guides are read and the required inputs are generated into a .cpp file (e.g. $ADL_HOME/EDR/VIIRS/imagery/cloudMask/src/AutoGeneratedProEdrViirsCM.cpp). So if you had only changed PRO_CMN_ADJ_CFG.xml, when you ran the executable it would still be using code that said the GRC was required. It would attempt to retrieve the GRC and fail. It would be possible to make the GRC optional in ProEdrViirsActiveFires_CFG.xml and ProEdrViirsCM_CFG.xml, but there is resistance to doing that in the operational system and we want to keep the ADL and operational config guides linked. (In our development environment, the ADL configs are a symbolic link to the operational configs.)

Regarding GRC and GTM, I spoke to our GTM developer about the issue and he said that as currently coded GTM requires the GRC input. He thought it would *probably* be possible to change some code and have GTM derive the info it needs from GRC from GEO instead. However, that change would need to be driven by customer need, which it sounds like you have. If you want to start pushing the change through the appropriate IDPS boards, the change may be possible.