About LUT update

[jhuangadl]

I am now having a new LUT file:

Code: Select all

VIIRS-AOT-LUT_npp_20130410000000Z_20110101000000Z_ee00000000000000Z_PS-1-O-CCR-xx-xxx-JPSS-DPA-002-PE_aot16_all_all-_all.bin

to replace
the old one:

Code: Select all

4d911423-951e1-0a4f180b-57e9382f.asc
4d911423-951e1-0a4f180b-57e9382f.VIIRS-AOT-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-3-PE-_devl_dev_all-_all.bin

My questions are:
1) Do I have to generate the corresponding *.asc file to run in ADL using the new LUT? (the new LUT is already in the binary format)
2) If yes to 1), how can I generate the *.asc file from the *.bin file? There are no such tools available in ADL, are there? I know I can genreate a Urid using the Uridcreater.exe, but do I have to manually put information for a *.asc file?
Thanks!
Jingfeng

[jhuangadl]

Also, I am having the following error when I attempted to run ADL using the new LUT:

Code: Select all

ProEdrViirsAerosolController failed to get cross granule inputs
ProCmnDataItem[17ProCmnAuxDataItem]::setDataVectorExisting(VIIRS-AOT-LUT) [0x3748600] ROOT PRO_FAIL wrong file size: mapped file size 49730128 != expected size 46622120 from file ProCmnDataItem.cpp, line 2721
ProCmnAlgorithm[ProEdrViirsAerosol]::handleInputQuery(NPP000424169869) [0x2d9e4a0] ROOT PRO_CROSSGRAN_FAIL Required input not available for Shortname: VIIRS-AOT-LUT , Granule ID:  Failure(s):  (VIIRS-AOT-LUT, ) from file ProCmnAlgorithm.cpp, line 7895

which seems to indicate that we also need to change the file size information somewhere else as well. Any additional help are deeply appreciated!
Thanks,
Jingfeng

I am now having a new LUT file:

Code: Select all

VIIRS-AOT-LUT_npp_20130410000000Z_20110101000000Z_ee00000000000000Z_PS-1-O-CCR-xx-xxx-JPSS-DPA-002-PE_aot16_all_all-_all.bin

to replace
the old one:

Code: Select all

4d911423-951e1-0a4f180b-57e9382f.asc
4d911423-951e1-0a4f180b-57e9382f.VIIRS-AOT-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-3-PE-_devl_dev_all-_all.bin

My questions are:
1) Do I have to generate the corresponding *.asc file to run in ADL using the new LUT? (the new LUT is already in the binary format)
2) If yes to 1), how can I generate the *.asc file from the *.bin file? There are no such tools available in ADL, are there? I know I can genreate a Urid using the Uridcreater.exe, but do I have to manually put information for a *.asc file?
Thanks!
Jingfeng

[bhenders]

Jingfeng,

You need to have an updated $ADL_HOME/xml/VIIRS/VIIRS_AOT_LUT.xml file to match your new LUT format wise to reflect the change in size. It's not clear to me if you've modified the AOT LUT or if this was done externally from you and you are just trying to run with it. Nonetheless ADL needs to have an AOT LUT product format that reflects the format of the new LUT. ADL compiles C++ structures from the XML product definition which is where the size of the item comes in. VIIRS product dictionary information is also auto generated from the XML such that the ADL framework knows the format and can provide service such as endian conversion on the input.

You'll need to get the updated product format XML or if you've instigated these changes you will need to use the ADL product format XML editor tool to modify the LUT format appropriate for your new version, this tool can be started by running: $ADL_HOME/script/xmled

Once you have a new XML file for the new LUT you will need to rebuild ADL to pick up the new format.

You will need to then insert the new LUT into your ADL input directory or DMS instance area.

This can be accomplished with the following script:

> $ADL_HOME/script/DMS/InsertIntoDms.plx -csn VIIRS-AOT-LUT -file NEWFILENAME -eff "2002-11-22 00:00:00.000000Z,2038-01-19 03:14:07.000000Z" -dir ADL_INPUT_DIRECTORY

This should allow you to run with this new input. I've given quite a few steps, please post back if I can help further or if you get stuck.

Thanks,

Bryan Henderson
Raytheon Company

[geoffc]

I have an xml description of a new VIIRS-CM-IP-AC file (which contains the field descriptions and values), is there an executable in ADL which can generate the binary VIIRS-CM-IP-AC file from the xml description?

Below is a snippet of the xml file...

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
 <!--
       DQM Threshold Check file / table example.
       author: Eric Ort, Raytheon - DQM - CO Springs, CO, 719-637-7093,
  -->
 
 <!DOCTYPE dqmtable SYSTEM "VIIRS-CM-IP-AC_npp.dtd">
    <!-- 
       HISTORY:
         DATE:        CCR#       AUTHOR          BUILD        DESCRIPTION
    
      09SEP2011      11-174      N. Baker       MX5           Baseline for the tunable coefficients per NGAS delivery
      03DEC2011      11-232      N. Baker       MX5.1         Update of night time coefficients per NGAS ECR ;
                                                              Added descriptions of tunable coefficients 
      13APR2012      12-382      W. Thomas      MX5.3         Post launch, 30-day spinup update of tunable coefficients 
      15JUN2012      DR4577      B. Iisager                   Add VCM_SNOWICE_POLAR_LAT
      27JUL2012      DR4793      B. Iisager                   Replace each water/day M7 threshold with
                                                              poly coeffs and a final adjustment value
      27JUL2012      DR4734      B. Iisager                   Tune out volcanic ash test for now set
                                                              VCM_AERO_COMP_OCEAN_M15M16BTD_MAX_THRESH to -5.0 from
                                                              -0.25
                     CCR 12-637  B. Iisager                   Update WD_M7_* coefs and corrs
                                                              WD_M7_SNGLNT_* coefs and corrs and
                                                              LN_M12_M16 coeffs
    -->
    <!--
       DQM Table declaration and mandatory type.
    -->
    <dqmtable type="Coefficient">
    <!--
       Superset of all attributes used by coefficients in this table.  Used in
     building the columns of any table used to view the coefficients.
    -->
    <columnlist>Description</columnlist>
   
     <!--
          Filter/data on this Coefficient table.
      'spacecraft'   - The spacecraft represented by this DQ Threshold XML file.
      'sensor'       - The sensor represented by this DQ Threshold XML file.
      'category'     - The category represented by this DQ Threshold XML file.
      'product'      - The product represented by this DQ Threshold XML file.
      'csname'       - the Collection Short Name for the spacecraft, sensor (etc.)
                           represented by this file.
     
      Each 'coeff' contains info representing 1 coefficient and consists of
      an attribute list which vary from file to file (product to product)
      but should be roughly the same amongst 'coeff' instances within the same
      file/product.  Each 'coeff' also contains fixed tags that are the same across
      all files/products for each 'coeff'.  They are:
      'id'           - The ID of this coefficient within this file/product.
      'name'         - The name of this coefficient within this file/product.
      'type'         - The type of this coefficient (eg. 'int', 'float', 'long', double').
      'value'        - The value of this coefficient in the given type.
      'sdigits'      - The significant digits that this value should be in.
      'min'          - The minimum value that the value can be.
      'max'          - The maximum value that the value can be.
 
     -->
 <coefficients category="IP" csname="VIIRS-CM-IP-AC" product="CM" sensor="VIIRS" spacecraft="NPP">
 
   <coeff Description="units=Kelvin; maximum M15 brightness temperature at which snow/ice may exist">
     <id>0</id>
     <name>snow_thresh_BTM15</name>
     <type>float</type>
     <value>275.0</value>
     <sdigits>5</sdigits>
     <min>260.0</min>
     <max>285.0</max>
   </coeff>

   ...

   </coefficients>
   </dqmtable>

[bhenders]

Geoff,

Please see the ADL Workshop slides that are named "11_ADL_Workshop_Ingest_MSD_Tool.pptx" which describes how to convert a Data Qality Monitoring (DQM) XML file into the internal binary representation.

I'll paste in relevant info from the slides where the example in the slides was Cloud Mask Coefficients:

IDPS Configurable Coefficients files are set by DQM xml files. These files are ingested through the ING MSD to create binary files to be input to the code.
When updating configurable coefficients for IDPS algorithms, the best way is to modify the XML file and use the ADL ING MSD code to create the new binary file.
Copies of the latest versions of the DQM xml file at the time of the ADL delivery can be found under $ADL_HOME/cfg/dqmtables. This is included in the idpsAlgorithms.tgz file and not part of the ADL Framework by default.

Run the sortMsdIntoLz.pl script on the modified xml file to move it to the landing zone. > ${ADL_HOME}/script/sortMsdIntoLz.pl -c $ADL_HOME/cfg/dqmtables/NPP/VIIRS/IP/CM/VIIRS-CM-IP-AC_npp_20120701000000Z_20120103030000Z_ee00000000000000Z_PS-1-O-CCR-12-476-JPSS-DPA-004-PE_noaa_all_all-_all.xml> ls $ADL_MSD_LZ_PATH/*
/vobs/IDPS-CAT3/ADL/data/MSD_LZ/ANC:/vobs/IDPS-CAT3/ADL/data/MSD_LZ/AUX:VIIRS-CM-IP-AC_npp_20120701000000Z_20120103030000Z_ee00000000000000Z_PS-1-O-CCR-12-476-JPSS-DPA-004-PE_noaa_all_all-_all.xml/vobs/IDPS-CAT3/ADL/data/MSD_LZ/TILED:/vobs/IDPS-CAT3/ADL/data/MSD_LZ/Temp:

Run the runMsd.pl script to convert the XML file into the Internal binary format. Temporal Interpolation does not need to run on configurable coefficients files.
> ${ADL_HOME}/script/runMsd.pl -m
Running ING MSD...
ING MSD completed successfully
All files from Landing Zone converted.
Number of newly converted output files: 2
Running ING MSD only. Skipping TI check...

[kbisanz]

Similar information is also available in the ADL User's Manual. Unfortunately the manuals were not able to be delivered as part of ADL 4.1. However they are available at https://jpss.ssec.wisc.edu/adl/download ... 1/manuals/. You want the file 121015 CDRL A032 UG60917-IDP-034A IDPS ADL SUM Part 2.pdf.

I believe section "5.9.8 Bringing New Externally Formatted Data into ADL" is what you want. It contains information very similar to what bhenders posted.

[jhuangadl]

Thank you all,
I believe my issue was with the rebuilding of the ADL4.1 so that ADL can recognize the new XML file and match the appropriate file size of the new LUT.
However, I am now having problems with the ADL rebuilding:

Code: Select all

[jhuang@rhw9103 ADL]$ ./buildAdl.csh
 Error: Expected to find bin/perl at $PERL_HOME (/data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/bin/perl)
 Error: Expected to find lib/libfftw3.a at $COTS_FFTW_HOME (/data/data003/CSPP/common/local/lib/libfftw3.a)
 Error: Expected to find g2clib-1.0.4/libg2c.a at $COTS_WGRIB2_HOME (/data/data003/CSPP/common/local/g2clib-1.0.4/libg2c.a)
 Environment errors exist.  Please correct them and rerun ./buildAdl.csh.

I can understand the first one is because of the perl installation. I have successfully installed perl by the following:

Code: Select all

[jhuang@rhw9103 support]$ ./install_user_perl_for_ADL4.1.sh 

However, I still have the following error:

Code: Select all

[jhuang@rhw9103 ADL]$ ./buildAdl.ksh
Error: Expected to find lib/libfftw3.a at $COTS_FFTW_HOME (/data/data003/CSPP/common/local/lib/libfftw3.a)
Error: Expected to find g2clib-1.0.4/libg2c.a at $COTS_WGRIB2_HOME (/data/data003/CSPP/common/local/g2clib-1.0.4/libg2c.a)
Environment errors exist.  Please correct them and rerun ./buildAdl.ksh.

I couldn't find the following two files at all:

Code: Select all

CSPP/common/local/lib/libfftw3.a
CSPP/common/local/g2clib-1.0.4/libg2c.a

This is so strange because we successfully built the ADL when we installed it locally, now it started complaining for more environment files for rebuilding. I must be seriously mistaken somewhere and I am stuck. Please please help.
Thank you!
Jingfeng

[kbisanz]

From this message:
Error: Expected to find lib/libfftw3.a at $COTS_FFTW_HOME (/data/data003/CSPP/common/local/lib/libfftw3.a)
Is there a /data directory in your installation?

If there is not a /data, it appears that it's expecting something to be added to the front of that path. Almost like $ADL_HOME isn't set or something.

I'm sure you know this by now, but the environment variables are set in $ADL_HOME/build/envSetup.ksh. So all the paths in there must be correct. Or at least that's how ADL works when it's shipped from Raytheon. I believe when it's packaged in UW's virtual appliance it's similar. But we are unable to install the virtual appliance (it's a long story) so I can't say for sure.

[jhuangadl]

Here is the puzzle:

If I build like the following, I will get the error message. This building process is what we did when we installed ADL I believe.

Code: Select all

[jhuang@rhw9103 ADL]$ ./buildAdl.ksh
Error: Expected to find lib/libfftw3.a at $COTS_FFTW_HOME (/data/data003/CSPP/common/local/lib/libfftw3.a)
Error: Expected to find g2clib-1.0.4/libg2c.a at $COTS_WGRIB2_HOME (/data/data003/CSPP/common/local/g2clib-1.0.4/libg2c.a)
Environment errors exist.  Please correct them and rerun ./buildAdl.ksh

Then I looked into the ADL manual and was advised to conduct the following rebuild:

Code: Select all

1. Execute “make src” in $ADL_HOME/xml/<sensor>, which is the directory containing the new or updated product profile XML.
2. Execute “make clean library” in $ADL_HOME/CMN/Utilities/Dictionary/Entries/<sensor>/src.
3. Execute “make clean library program” in $ADL_HOME/path/to/algorithm/src.
a. An updated pointer structure will appear in AutoGenerated<algorithm>.h.

It worked out for the second approach. What are the difference between the above two approaches?

I am giving a try with the new LUT, but I am getting segmentation fault:

Code: Select all

8106 Segmentation fault      (core dumped) ${ADL_HOME}/bin/ProEdrViirsAerosolController.exe ${ADL_HOME}/cfg/dynamic/withMetadata/ProEdrViirsAerosolControllerLwFile.xml 

I think I am missing some important steps. How can I make:

Code: Select all

VIIRS-AOT-LUT_npp_20130410000000Z_20110101000000Z_ee00000000000000Z_PS-1-O-CCR-xx-xxx-JPSS-DPA-002-PE_aot16_all_all-_all

to be like the following:

Code: Select all

4d911423-951e1-0a4f180b-57e9382f.asc
4d911423-951e1-0a4f180b-57e9382f.VIIRS-AOT-LUT_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_PS-1-D-NPP-3-PE-_devl_dev_all-_all.bin

with *asc file associated with it???
Thanks!!!

[jhuangadl]

I checked through bryant's reply again and tried the following:

Code: Select all

[jhuang@rhw9103 jhuang]$ $ADL_HOME/script/DMS/InsertIntoDms.plx -csn VIIRS-AOT-LUT -file VIIRS-AOT-LUT_npp_20130410000000Z_20110101000000Z_ee00000000000000Z_PS-1-O-CCR-xx-xxx-JPSS-DPA-002-PE_aot16_all_all-_all.bin -eff "2013-04-10 00:00:00.000000Z,2038-01-19 03:14:07.000000Z" -dir /data/data003/jhuang/anc_static/

But encountered the following errors:

Code: Select all

Bareword found where operator expected at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8_heavy.pl line 21, near "s/[-\s_]//rg"
syntax error at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8_heavy.pl line 21, near "s/[-\s_]//rg"
BEGIN not safe after errors--compilation aborted at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8_heavy.pl line 147.
Compilation failed in require at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/utf8.pm line 17.
BEGIN failed--compilation aborted at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/constant.pm line 45.
Compilation failed in require at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/Getopt/Long.pm line 208.
BEGIN failed--compilation aborted at /data/data003/CSPP/common/COTS/perl/perls/perl-5.15.9/lib/5.15.9/Getopt/Long.pm line 208.
Compilation failed in require at /data/data003/CSPP/ADL/script/DMS/InsertIntoDms.plx line 93.
BEGIN failed--compilation aborted at /data/data003/CSPP/ADL/script/DMS/InsertIntoDms.plx line 93.

Still trying to dig around.