How to toggle endianness conversion for VIIRS?

geoffc · Post by **geoffc** » Fri Mar 29, 2013 2:39 pm

While I have been testing the VIIRS Cloud Mask and Aerosol controllers, I have been using alternately the canned test data which comes with ADL (which is all Big Endian), and SDR, ANC and GridIP files generated locally outside of ADL (all Little Endian). I expected that ADL would convert the BE test data to LE when running on my LE x86 test machine, but it appears that the endianness conversion is also being performed (I assume erroneously) on LE data. This has resulted in the float ANC data (only affected datatype as far as I can tell) having the BE representation of the proper float32 value (-495925376 as opposed to -999.700012 for a bow-tie pixel) once the ANC data has been read into the appropriate data structure in the controller. For reference, I plotted the BE and LE ancillary data in such a way that I have to specify the endianness of the input files: all plots came out as expected, so I am confident the BE and LE ancillary files have the endianness that I think they do.

I ran the VIIRS cloud mask and Aerosol controllers through a debugger, and printed out the first pixel of each of the SDR, ANC and GridIP data arrays once they had been ingested into the controller. Below we show the debugger output for the BE VCM input data on the left, and the LE VCM input data on the right...

Code: Select all

Breakpoint 1, ProEdrViirsCM::doProcessing (this=0x2aab22ee5010) at ProEdrViirsCM.cpp:734

(gdb) print *(vcmDataType_.lat+0) = 1.23894072                  (gdb) print *(vcmDataType_.lat+0) = 0.865942657
(gdb) print *(vcmDataType_.lon+0) = 1.93343997                  (gdb) print *(vcmDataType_.lon+0) = -0.937229097
(gdb) print *(vcmDataType_.satzen+0) = 1.22850406               (gdb) print *(vcmDataType_.satzen+0) = 1.22519171
(gdb) print *(vcmDataType_.sunzen+0) = 1.63769233               (gdb) print *(vcmDataType_.sunzen+0) = 1.32874489
(gdb) print *(vcmDataType_.satazm+0) = -1.30861139              (gdb) print *(vcmDataType_.satazm+0) = -1.51278567
(gdb) print *(vcmDataType_.sunazm+0) = -2.51388383              (gdb) print *(vcmDataType_.sunazm+0) = -2.54166651

(gdb) print *(vcmDataType_.reflM1 + 0) = -999.700012            (gdb) print *(vcmDataType_.reflM1 + 0) = -999.700012
(gdb) print *(vcmDataType_.reflM4 + 0) = -999.700012            (gdb) print *(vcmDataType_.reflM4 + 0) = -999.700012
(gdb) print *(vcmDataType_.reflM5 + 0)  = -999.700012           (gdb) print *(vcmDataType_.reflM5 + 0) = -999.700012
(gdb) print *(vcmDataType_.reflM7 + 0)  = -999.700012           (gdb) print *(vcmDataType_.reflM7 + 0)  = -999.700012
(gdb) print *(vcmDataType_.reflM9 + 0)  = -999.700012           (gdb) print *(vcmDataType_.reflM9 + 0)  = -999.700012
(gdb) print *(vcmDataType_.reflM10 + 0)  = -999.700012          (gdb) print *(vcmDataType_.reflM10 + 0)  = -999.700012
(gdb) print *(vcmDataType_.reflM11 + 0)  = -999.700012          (gdb) print *(vcmDataType_.reflM11 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btM12 + 0)  = -999.700012            (gdb) print *(vcmDataType_.btM12 + 0)  = -999.700012
(gdb) print *(vcmDataType_.radM12 + 0)  = -999.700012           (gdb) print *(vcmDataType_.radM12 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btM13 + 0)  = -999.700012            (gdb) print *(vcmDataType_.btM13 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btM14 + 0)  = -999.700012            (gdb) print *(vcmDataType_.btM14 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btM15 + 0)  = -999.700012            (gdb) print *(vcmDataType_.btM15 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btM16 + 0)  = -999.700012            (gdb) print *(vcmDataType_.btM16 + 0)  = -999.700012

(gdb) print *(vcmDataType_.reflI1 + 0)  = -999.700012           (gdb) print *(vcmDataType_.reflI1 + 0)  = -999.700012
(gdb) print *(vcmDataType_.reflI2 + 0)  = -999.700012           (gdb) print *(vcmDataType_.reflI2 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btI4 + 0)  = -999.700012             (gdb) print *(vcmDataType_.btI4 + 0)  = -999.700012
(gdb) print *(vcmDataType_.btI5 + 0)  = -999.700012             (gdb) print *(vcmDataType_.btI5 + 0)  = -999.700012

(gdb) print *(vcmDataType_.tpwIP + 0)  = -999.700012            (gdb) print *(vcmDataType_.tpwIP + 0)  = -495925376
(gdb) print *(vcmDataType_.sfc_temp + 0)  = -999.700012         (gdb) print *(vcmDataType_.sfc_temp + 0)  = -495925376
(gdb) print *(vcmDataType_.windSpeed + 0)  = -999.700012        (gdb) print *(vcmDataType_.windSpeed + 0)  = -495925376
(gdb) print *(vcmDataType_.toc_ndvi + 0)  = -999.700012         (gdb) print *(vcmDataType_.toc_ndvi + 0)  = -495925376

(gdb) print *(vcmDataType_.qstLwm + 0)  = 253 '\375'            (gdb) print *(vcmDataType_.qstLwm + 0)  = 253 '\375'
(gdb) print *(vcmDataType_.snow + 0)  = 253 '\375'              (gdb) print *(vcmDataType_.snow + 0)  = 253 '\375'
(gdb) print *(vcmDataType_.terrainHeight + 0)  = -997           (gdb) print *(vcmDataType_.terrainHeight + 0)  = 7420

We can see that for the LE data, the float ancillary data (tpwIP, sfc_temp, windSpeed, toc_ndvi) seem to be clearly affected. For the LE terrainHeight, 7420 is the BE representation for -996 for an int16 datatype, so I think terrainHeight is affected also.

We have similarly for the aerosol controller...

Code: Select all

Breakpoint 1, ProEdrViirsAerosol::AOT_main (this=0x2aab1c29bbe0) at ProEdrViirsAerosol.cpp:4255

(gdb) print *(sdr_data.lat+0) = 1.24410117                                       (gdb) print *(sdr_data.lat+0) = 0.530207098
(gdb) print *(sdr_data.lon+0) = 1.9371506                                        (gdb) print *(sdr_data.lon+0) = -0.917341113
(gdb) print *(sdr_data.senzen+0) = 1.22847033                                    (gdb) print *(sdr_data.senzen+0) = 1.22413874
(gdb) print *(sdr_data.solzen+0) = 1.64264297                                    (gdb) print *(sdr_data.solzen+0) = 1.05592728
(gdb) print *(sdr_data.senazi+0) = -1.3033421                                    (gdb) print *(sdr_data.senazi+0) = -1.59942055
(gdb) print *(sdr_data.solazi+0) = -2.50986814                                   (gdb) print *(sdr_data.solazi+0) = -2.46534491

(gdb) print *(sdr_data.reflM1 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM1 + 0) = -999.700012
(gdb) print *(sdr_data.reflM2 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM2 + 0) = -999.700012
(gdb) print *(sdr_data.reflM3 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM3 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM4 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM4 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM5 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM5 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM6 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM6 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM7 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM7 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM7 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM7 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM8 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM8 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM9 + 0) = -999.700012                                 (gdb) print *(sdr_data.reflM9 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM10 + 0) = -999.700012                                (gdb) print *(sdr_data.reflM10 + 0)  = -999.700012
(gdb) print *(sdr_data.reflM11 + 0) = -999.700012                                (gdb) print *(sdr_data.reflM11 + 0)  = -999.700012
(gdb) print *(sdr_data.btM12 + 0) = -999.700012                                  (gdb) print *(sdr_data.btM12 + 0)  = -999.700012
(gdb) print *(sdr_data.btM15 + 0) = -999.700012                                  (gdb) print *(sdr_data.btM15 + 0)  = -999.700012
(gdb) print *(sdr_data.btM16 + 0) = -999.700012                                  (gdb) print *(sdr_data.btM16 + 0)  = -999.700012

(gdb) print *(mdl_data.windspd + 0) = -999.700012                                (gdb) print *(mdl_data.windspd + 0)  = -495925376
(gdb) print *(mdl_data.winddir + 0) = -999.700012                                (gdb) print *(mdl_data.winddir + 0)  = -495925376
(gdb) print *(mdl_data.pw + 0) = -999.700012                                     (gdb) print *(mdl_data.pw + 0)  = -495925376
(gdb) print *(mdl_data.sairtemp + 0) = -999.700012                               (gdb) print *(mdl_data.sairtemp + 0)  = -495925376
(gdb) print *(mdl_data.ozconc + 0) = -999.700012                                 (gdb) print *(mdl_data.ozconc + 0)  = -495925376
(gdb) print *(mdl_data.psl + 0) = -999.700012                                    (gdb) print *(mdl_data.psl + 0)  = -495925376

(gdb) print memoryPtrData_.aotLutPtr->aot[0] = 0.00999999978                     (gdb) print memoryPtrData_.aotLutPtr->aot[0]  = 0.00999999978
(gdb) print memoryPtrData_.aotLutPtr->szen[0] = 0                                (gdb) print memoryPtrData_.aotLutPtr->szen[0]  = 0
(gdb) print memoryPtrData_.aotLutPtr->vzen[0] = 0                                (gdb) print memoryPtrData_.aotLutPtr->vzen[0]  = 0
(gdb) print memoryPtrData_.aotLutPtr->scat_ang_incr = 0.069813170079773182       (gdb) print memoryPtrData_.aotLutPtr->scat_ang_incr = 0.069813170079773182
(gdb) print memoryPtrData_.aotLutPtr->scat_ang_val[0] = 0                        (gdb) print memoryPtrData_.aotLutPtr->scat_ang_val[0]  = 0
(gdb) print memoryPtrData_.aotLutPtr->trans[0][0][0][0] = 0.859589994            (gdb) print memoryPtrData_.aotLutPtr->trans[0][0][0][0]  = 0.859589994
(gdb) print memoryPtrData_.aotLutPtr->albedo[0][0][0] = 0.218370005              (gdb) print memoryPtrData_.aotLutPtr->albedo[0][0][0]  = 0.218370005
(gdb) print memoryPtrData_.aotLutPtr->atau[0][0][0] = 1.72800004                 (gdb) print memoryPtrData_.aotLutPtr->atau[0][0][0]  = 1.72800004
(gdb) print memoryPtrData_.aotLutPtr->reflec[0][0][0][0] = 0.122680001           (gdb) print memoryPtrData_.aotLutPtr->reflec[0][0][0][0]  = 0.122680001

Again the float ANC data (windspd, winddir, pw, sairtemp, ozconc, psl) appears to have the wrong endianness. Note that for the aerosol case I have also included some values from the various LUTS. These are always BE files, whether I am using BE or LE SDR/ANC/GriIP files. We can see that the LUT values seem to be OK in both cases.

To try and figure out what ADL was doing with regards to endian conversions, I grepped the log files looking for instances where ADL found it necessary to convert. For the VCM (BE left, LE right)...

Code: Select all

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I1-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I1-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I2-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I2-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-I5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M10-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M10-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M10-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M11-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M11-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M12-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M12-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M12-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M13-SDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M13-SDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M13-SDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M14-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M14-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M14-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M15-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M15-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M15-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M16-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M16-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M16-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M1-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M1-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M7-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M7-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M7-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M9-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M9-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC                                  DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC                                  DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC                                  DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC-TC                               DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC-TC                               DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC-TC                               DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-GRC-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO-TC
`
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AF-EDR-AC-Int                            DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AF-EDR-AC-Int
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AF-EDR-DQTT-Int                          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AF-EDR-DQTT-Int
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-CM-IP-AC-Int                             DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-CM-IP-AC-Int

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran                   DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran                   DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran                   DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Surf-Ht-Mod-Gran                     DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Surf-Ht-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Surf-Ht-Mod-Gran                     DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Surf-Ht-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Surf-Ht-Mod-Gran                     DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Surf-Ht-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran                 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran                 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran                 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran                  DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran                  DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran                  DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Nbar-Ndvi-Mod-Gran          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Nbar-Ndvi-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Nbar-Ndvi-Mod-Gran          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Nbar-Ndvi-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Nbar-Ndvi-Mod-Gran          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Nbar-Ndvi-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran            DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran            DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran            DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran     DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran     DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran     DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-GridIP-VIIRS-Snow-Ice-Cover-Mod-Gran

For the LE case on the right the VIIRS-MOD-GRC* are converted because I am using the BE VIIRS-MOD-GRC* files as placeholders (they're not used to do any granulation). And I also use BE LUTS . The VIIRS-ANC* and VIIRS-GridIP-VIIRS* are however LE and should not be converted. The SDRs are (correctly) not converted as they are unpacked from HDF5 as LE files.

And for the AOT...

Code: Select all

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M10-FSDR                                
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M10-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M10-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M11-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M11-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M12-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M12-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M12-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M15-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M15-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M15-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M16-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M16-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M16-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M1-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M1-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M2-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M2-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M3-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M3-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M4-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M5-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M6-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M6-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M7-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M7-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M7-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M8-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M8-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M8-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M9-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-M9-FSDR
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO-TC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-MOD-RGEO-TC

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-CM-IP
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-CM-IP
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-CM-IP

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for AOT-ANC                             DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for AOT-ANC
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-Aeros-EDR-AC-Int              DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-Aeros-EDR-AC-Int
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-Aeros-EDR-DQTT-Int            DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-Aeros-EDR-DQTT-Int
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AOT-LUT                       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AOT-LUT
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AOT-Sunglint-LUT              DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-AOT-Sunglint-LUT
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-SusMat-EDR-DQTT-Int           DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-SusMat-EDR-DQTT-Int

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Optical-Depth-Mod-Gran    DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Optical-Depth-Mod-Gran

 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran        DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran        DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran        DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Preci-Wtr-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Press-Surf-Mod-Gran       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Press-Surf-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Press-Surf-Mod-Gran       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Press-Surf-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Press-Surf-Mod-Gran       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Press-Surf-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran      DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran      DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran      DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Temp-Surf2M-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Tot-Col-Mod-Gran          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Tot-Col-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Tot-Col-Mod-Gran          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Tot-Col-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Tot-Col-Mod-Gran          DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Tot-Col-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Direction-Mod-Gran   DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Direction-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Direction-Mod-Gran   DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Direction-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Direction-Mod-Gran   DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Direction-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran
 DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran       DBG_HIGH ProCmnInputItem.cpp|955|Converted endianness of data for VIIRS-ANC-Wind-Speed-Mod-Gran

Here we see that for the LE case on the right, we are again using the BE LUTS, so they are converted. However, the VIIRS-ANC-* files are all LE and should not be converted.

Of course, all of the above is predicated on the assumption that if all of the inputs have the same endianness as the test machine (Little Endian in my case), that there should be no conversions done. Is this the case.

I tried changing the "DataEndianType" for the various ancillary inputs specified in ProEdrViirsCM_CFG.xml and ProEdrViirsAerosol_CFG.xml, but that did not appear to have any effect. Are there any other ways to control what input files are subject to endianness conversion?

geoffc · Post by **geoffc** » Fri Mar 29, 2013 3:36 pm

I just found a reference to specifying the endianness of the inputs in the associated asc files by adding "("ByteOrder" STRING EQ "LE")" ...

viewtopic.php?f=32&t=133&p=425&hilit=Se ... 80%A6#p419

Will reply soon if this fixes my problem. Chime in if you have an opinion...

bhenders · Post by **bhenders** » Mon Apr 01, 2013 8:16 am

Geoff,

The ByteOrder metadata denotes the endianness of an input when running an algorithm in ADL with metadata on. The DataEndainType in the algorithm config files is only used when running in no metadata mode in ADL to denote the endianness of the specified files.

Getting the ByteOrder metadata set correctly for the endianness of any data you create is very important and the framework uses that in it's determination of whether to do endian conversion on the data.

Thanks,

Bryan Henderson
Raytheon Company

geoffc · Post by **geoffc** » Mon Apr 01, 2013 1:41 pm

Thanks Bryan,
I was going in circles with the *CFG.xml endian stuff, being able to specify the endianness on the run in the metadata is a big help, as I want my package to be able to generate BE or LE ancillary. After adding the "ByteOrder" directive to the ancillary metadata (I am using "withMetadata"), the AOT ran to completion. However, I am still getting...

Code: Select all

Breakpoint 1, ProEdrViirsAerosol::AOT_main (this=0x2aab1c29bbe0) at ProEdrViirsAerosol.cpp:4255

(gdb) print *(mdl_data.windspd + 0)  = -495925376
(gdb) print *(mdl_data.winddir + 0)  = -495925376
(gdb) print *(mdl_data.pw + 0)  = -495925376
(gdb) print *(mdl_data.sairtemp + 0)  = -495925376
(gdb) print *(mdl_data.ozconc + 0)  = -495925376
(gdb) print *(mdl_data.psl + 0)  = -495925376

...at line 4255 of ProEdrViirsAerosol.cpp, even where the AOT runs successfully. I need to check the ancillary values later in the algorithm, especially where there are some sanity checks for ancillary values, to convince myself whether or not I'm running the AOT properly.

I checked the log files for endian conversions, the only things that are flagged as being converted are the LUTS, which I know are BE, so that seems to be in order.

I'm still getting the odd crash due to bad input to the molecular optical depth code, but now that I know the endianness issue is sorted this shouldn't be difficult to track down.

Thanks for the replies, I'm marking this as resolved.

SSEC forums

How to toggle endianness conversion for VIIRS?

How to toggle endianness conversion for VIIRS?

Re: How to toggle endianness conversion for VIIRS?

Re: How to toggle endianness conversion for VIIRS?

Re: How to toggle endianness conversion for VIIRS? [RESOLVED