About LUT update

[bhenders]

The basic difference between the two approaches is that the buildAdl.ksh script includes the $ADL_HOME/script/checkEnv.pl script which audits environment variables and COTS installations. It was expecting to find libfftw3.a and g2clib-1.0.4/libg2c.a at the locations specified. The second approach does not do this environment audit. Since, you were able to get ADL to build you must have these COTS packages installed but potentially with different versions or paths. The $ADL_HOME/script/checkEnv.pl may need to be tweaked for this installation. I think I saw that this install was a virtual appliance, if so this issue may need to go back to the University of Wisconsin to address, since at Raytheon we are unable to download the ADL virtual appliance.

Thanks,

Bryan Henderson
Raytheon Company

Here is the puzzle:

If I build like the following, I will get the error message. This building process is what we did when we installed ADL I believe.

Code: Select all

[jhuang@rhw9103 ADL]$ ./buildAdl.ksh
Error: Expected to find lib/libfftw3.a at $COTS_FFTW_HOME (/data/data003/CSPP/common/local/lib/libfftw3.a)
Error: Expected to find g2clib-1.0.4/libg2c.a at $COTS_WGRIB2_HOME (/data/data003/CSPP/common/local/g2clib-1.0.4/libg2c.a)
Environment errors exist.  Please correct them and rerun ./buildAdl.ksh

Then I looked into the ADL manual and was advised to conduct the following rebuild:

Code: Select all

1. Execute “make src” in $ADL_HOME/xml/<sensor>, which is the directory containing the new or updated product profile XML.
2. Execute “make clean library” in $ADL_HOME/CMN/Utilities/Dictionary/Entries/<sensor>/src.
3. Execute “make clean library program” in $ADL_HOME/path/to/algorithm/src.
a. An updated pointer structure will appear in AutoGenerated<algorithm>.h.

It worked out for the second approach. What are the difference between the above two approaches?

I am giving a try with the new LUT, but I am getting segmentation fault:

Code: Select all

8106 Segmentation fault      (core dumped) ${ADL_HOME}/bin/ProEdrViirsAerosolController.exe ${ADL_HOME}/cfg/dynamic/withMetadata/ProEdrViirsAerosolControllerLwFile.xml 

I think I am missing some important steps. How can I make:

Code: Select all

VIIRS-AOT-LUT_npp_20130410000000Z_20110101000000Z_ee00000000000000Z_PS-1-O-CCR-xx-xxx-JPSS-DPA-002-PE_aot16_all_all-_all

to be like the following:

Code: Select all

4d911423-951e1-0a4f180b-57e9382f.asc
4d911423-951e1-0a4f180b-57e9382f.VIIRS-AOT-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-3-PE-_devl_dev_all-_all.bin

with *asc file associated with it???
Thanks!!!

[bhenders]

Jingfeng,

I'll have to talk to Kevin in the morning about the perl error you are getting on the InsertIntoDms.plx script, this may be a side effect of the perl version in the virtual appliance. In the mean time, you can just update the file name in the existing VIIRS-AOT-LUT ascii file to point to the new file name, leaving everything else the same. You might want to make a copy of the file before editing it by hand.

If you've updated the product XML to reflect the new size and recompiled ADL, you should then be able to retrieve the updated LUT by merely swapping the file names in the current metadata file.

Thanks,

Bryan Henderson
Raytheon Company

[kbisanz]

Regarding the checkEnv.pl script that bhenders mentioned. It is located at $ADL_HOME/script/checkEnv.pl. The purpose of the script is to do a quick sanity check of the environment before building ADL. We found that many ADL build issues were related to COTS not being installed in the expected locations. What checkEnv.pl does is for each environment variable, it tries to do some sort of check to make sure it's set to a reasonable value.

Consider these lines from checkEnv.pl:

Code: Select all

# Hash with environment variables that are directories with the key and a
# file that should be found in the directory as the value.
my %dirEnvVarsWithFiles = (
ADL_HOME => 'buildAdl.csh',
DCONFIG => 'ADL_CFG.xml',
DSTATICDATA => 'IETTime.dat',
COTS_BOOST_INC => 'boost/lexical_cast.hpp',
COTS_HDF5_HOME => 'lib/libhdf5.a',
COTS_LOG4CPLUS_HOME => 'lib/liblog4cplus.a',
COTS_WGRIB2_HOME => 'g2clib-1.0.4/libg2c.a',
COTS_XER_CPP_HOME => 'include/xercesc/dom/DOMDocument.hpp',
PERL_HOME => 'bin/perl',
JAVA_HOME => 'bin/javac',
ANT_HOME => 'bin/ant',
COTS_BEANUTILS_HOME => 'commons-beanutils.jar',
COTS_DIGESTER_HOME => 'commons-digester.jar',
COTS_JAVA_JNI_INC => 'jni.h',
COTS_SWINGX_HOME => 'swingx-1.6.jar',
XALAN_JAVA_HOME => 'xalan.jar',
XERCES_JAVA_HOME => 'xercesImpl.jar',
COTS_BLAS_LIB_LOC => 'libblas.a',
COTS_DCDFLIB_LIB_LOC => 'libdcdflib.a',
COTS_FFTW_HOME => 'lib/libfftw3.a',
COTS_LAPACK_LIB_LOC => 'liblapack.a',
COTS_PPPACK_LIB_LOC => 'libpppack.a',
);

Each of those is a key value pair. For example, COTS_WGRIB2_HOME is an environment variable in ADL. And "g2clib-1.0.4/libg2c.a" should be a valid path under $COTS_WGRIB2_HOME. Basically, "$COTS_WGRIB2_HOME/g2clib-1.0.4/libg2c.a" should be a valid path. The script only checks the files listed above, so there are obviously ways to fool the script.

The two files it looks like you had issues with were:

Code: Select all

[jhuang@rhw9103 ADL]$ ./buildAdl.ksh
Error: Expected to find lib/libfftw3.a at $COTS_FFTW_HOME (/data/data003/CSPP/common/local/lib/libfftw3.a)
Error: Expected to find g2clib-1.0.4/libg2c.a at $COTS_WGRIB2_HOME (/data/data003/CSPP/common/local/g2clib-1.0.4/libg2c.a)
Environment errors exist.  Please correct them and rerun ./buildAdl.ksh

To fix the issue, I would recommend the following:
1. Verify that the proper location of $COTS_FFTW_HOME is /data/data003/CSPP/common/local and the proper location of $COTS_WGRIB2_HOME is /data/data003/CSPP/common/local.
2. Then verify that lib/libfftw3.a exists under /data/data003/CSPP/common/local and that g2clib-1.0.4/libg2c.a exists under /data/data003/CSPP/common/local. If you find something like g2clib-1.0.5/libg2c.a for example, you would then need to update the checkEnv.pl script to look for the correct file. Note that any changes in versions may not have been tested and aren't guaranteed to work.

[jhuangadl]

Dear Bryan and Kevin,

Thank you very much for following this up for us.

1. We are using a native installed ADL, but not a Virtual Appliance.
2. I tweaked the *.asc and the LUT file, but encountered segmentation fault. I consulted Sid Jackson and was told that aerosol LUT update with more elements in the LUT files will require an entire rebuild of ADL. Therefore, my way of rebuilding the XML files only will cause segmentation fault.
3. This means I have to rebuild from the first approach, that is buildAdl.ksh. I have to find the two files that went missing.
4. For the first file, I only found libfftw3.la instead of libfftw3.a under path /data/data003/CSPP/common/local/lib --- are they the same or replacable?
5. For the second file, I couldn't find the installation of this package at all. Any chance we can install this package only, rather than rebuilding COTS?

Thank you!

Jingfeng

[bhenders]

Jingfeng,

Thanks for letting us know you are using a native installed ADL, I think we were thrown off by the presence of the CSPP in your paths. We still wonder how you got your COTS installed so maybe you can describe how you created your native install. Especially if we continue to have problems that seem COTS related.

Kevin and I are unsure of the issues that you are getting with the InsertIntoDms.ksh script, but believe it is related to the perl install somehow, or maybe the usage of a newer version of perl than we tested with.

It seems that you are missing the fftw and wgrib2 COTS libraries from your install. I believe that you may get ADL to compile successfully without them as only a few executables need fftw (CrIS SDR) and wgrib2 (ING MSD). FFTW can be found at UW's COTS page, however, I don't see WGRIB2 there:

https://jpss-adl-wiki.ssec.wisc.edu/med ... osted_COTS

What I would do as a quick test would be to comment out the following line in the buildAdl.ksh script, so the environment checker script doesn't even run. You should then be able to get a compile going and determine if the missing libraries manifest themselves in compile or run-time issues. If it's only run-time issues, you may not care as you are not executing executables that rely on those COTS libraries.

# Call the environment checker script to see if our environment is set up
# correctly. A non-0 return value indicates an error.
#$ADL_HOME/script/checkEnv.pl
#if [ $? != 0 ]
#then
# echo "Environment errors exist. Please correct them and rerun $0."
# exit 1
#fi

[jhuangadl]

Many thanks, Bryan,
I commented out those lines as you suggested, and the rebuilding is currently running. So far so good. I will post back if I continue to have problems.
Thank you for all the timely help!
Jingfneg

[jhuangadl]

Good new! After a successful rebuild, ADL is now running aerosol code with a new LUT.
Thank you, everyone!