issues related to ADL_Packer.exe

[ljiang]

ADLPacker.exe program assumes the byte-order of the binary which it tries to convert to be the native format of the machine/OS. On our Linux machines, the native format is "little-endian". Therefore, the ADLPacker.exe will always assume the binary file to be in "little-endian" format. So if we use ADLPacker.exe on a input binary file provided by ADL or operational IDPS, which is big-endian, the resulting HDF5 file will be wrong.

Should the ADL_Packer.exe generate correct HDF5 according to "ByteOrder" in the meta file?

[kbisanz]

You are correct that the ADL_Packer.exe assumes a binary is native to the current platform. So, you are also correct that there will be problems if you try to pack a big-endian product on a little endian machine. We are currently investigating a solution for that.

As a work around for now, I would recommend that you use the AdlFileEndianConverter.exe to convert the endianness to native (little endian in this case) and then use the ADL_Packer.exe to create an HDF file. AdlFileEndianConverter.exe has the following usage statement:

Code: Select all

Usage: ./AdlFileEndianConverter.exe <input file name>  <group name> <sensor name> <output file name>
Required Arguments:
    -- <input file name> : The name of the file on which
       endian conversion will be performed
    -- <group name> : The group name of the product, as
       specified by <ProductGroupName> in product profile XML
    -- <sensor name> : Name of the sensor to retrieve the
       dictionary entry from.  The sensor should be one of:
       Science Template ANC CRIMSS OMPS-NP OMPS-TC VIIRS
    -- <output file name> : The name of the file where the
       endian converted data should be written

For example, I converted the big endian IST truth data to little endian by using the following steps. Note that "/vobs/IDPS-CAT3/ADL/bin > " is my shell prompt.

Make a temporary directory and copy in the big endian outputs

Code: Select all

/vobs/IDPS-CAT3/ADL/bin > mkdir test
/vobs/IDPS-CAT3/ADL/bin > cp $ADL_HOME/data/output/withMetadata/ProEdrViirsIstOutputs/4e42dce3-3f4b3-007f0100-09bf2116* test
/vobs/IDPS-CAT3/ADL/bin > cd test
/vobs/IDPS-CAT3/ADL/bin/test > ls
4e42dce3-3f4b3-007f0100-09bf2116.asc
4e42dce3-3f4b3-007f0100-09bf2116.VIIRS-IST-EDR.BE

Run the file endian convertor. See section "3.2.1.5.6.5 FileEndianConverter" in User Manual Part 2 for more information, including why "SCALED" was appended to the group name when using AdlFileEndianConverter.exe.

Code: Select all

/vobs/IDPS-CAT3/ADL/bin/test > $ADL_HOME/bin/AdlFileEndianConverter.exe 4e42dce3-3f4b3-007f0100-09bf2116.VIIRS-IST-EDR.BE  VIIRS_IST_EDRSCALED VIIRS temp.bin
LogLog: Configurator file key inf_util_dbg_cfg_file not specified
LogLog: Configurator class key inf_util_dbg_cfg_class not specified
LogLog: Configurator aborted
Conversion successful
/vobs/IDPS-CAT3/ADL/bin/test > mv temp.bin 4e42dce3-3f4b3-007f0100-09bf2116.VIIRS-IST-EDR
/vobs/IDPS-CAT3/ADL/bin/test > ls
4e42dce3-3f4b3-007f0100-09bf2116.asc
4e42dce3-3f4b3-007f0100-09bf2116.VIIRS-IST-EDR
4e42dce3-3f4b3-007f0100-09bf2116.VIIRS-IST-EDR.BE

Edit the .asc file to remove the ".BE" from the N_Collection_Short_Name, change the ByteOrder value, and update the file path.

Code: Select all

/vobs/IDPS-CAT3/ADL/bin/test > vim 4e42dce3-3f4b3-007f0100-09bf2116.asc
/vobs/IDPS-CAT3/ADL/bin/test > grep N_Collection_Short_Name 4e42dce3-3f4b3-007f0100-09bf2116.asc
  ("N_Collection_Short_Name" STRING EQ "VIIRS-IST-EDR")
/vobs/IDPS-CAT3/ADL/bin/test > grep ByteOrder 4e42dce3-3f4b3-007f0100-09bf2116.asc
  ("ByteOrder" STRING EQ "LE")
/vobs/IDPS-CAT3/ADL/bin/test > grep FILE$ 4e42dce3-3f4b3-007f0100-09bf2116.asc
  ("/vobs/IDPS-CAT3/ADL/data/output/withMetadata/ProEdrViirsIstOutputs/4e42dce3-3f4b3-007f0100-09bf2116.VIIRS-IST-EDR" FILE

Run the HDF packer to pack it in HDF. 65533 is the proper fill for onboard pixel trim for UInt16 values.

Code: Select all

/vobs/IDPS-CAT3/ADL/bin/test > export INFTK_DM_ROOT=$ADL_HOME/bin/test
/vobs/IDPS-CAT3/ADL/bin/test > cd ..
/vobs/IDPS-CAT3/ADL/bin > ./ADL_Packer.exe 4e42dce3-3f4b3-007f0100-09bf2116  $ADL_HOME/bin/test/ $ADL_HOME/cfg/DDS_DDSUTIL_GuideList.cfg
./ADL_Packer.exe: Finished
/vobs/IDPS-CAT3/ADL/bin > $COTS_HDF5_HOME/bin/h5dump test/VISTO_npp_d20030125_t0722288_e0723533_b00014_-_devl_dev.h5 | head -100 | tail
            (0,24): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,32): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,40): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,48): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,56): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,64): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,72): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,80): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,88): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
            (0,96): 65533, 65533, 65533, 65533, 65533, 65533, 65533, 65533,
/vobs/IDPS-CAT3/ADL/bin >

It should be noted that logging is currently broken for AdlFileEndianConverter.exe. If AdlFileEndianConverter.exe encounters an error while running, it will output the message "ERROR: conversion of <name> was not successful. See the debug log in the current directory.". However, the log file is not created. The logging issue will be fixed with the ADL 3.1 release. If necessary, all DEBUG messages in $ADL_HOME/CMN/Utilities/FileEndianConverter/src could be converted to cout's.