Weird issue in xml/VIIRS/VIIRS_LSA_IP_AC_Int.xml

[tangyouhua]

Recently I encountered a weird issue of xmled when changing LUT in
$ADL_HOME/xml/VIIRS/VIIRS_LSA_IP_AC_Int.xml

It is original definition is

struct IngMsdCoefficients_ViirsSurfAlbedoStruct
{
Float32 SnowThreshold;
Float32 NDVI_Threshold;
Int64 NumBins_SolarZenith;
Float32 BinSize_SolarZenith;
UInt8 implicit_pad0[PAD_BYTES_4];
Int64 NumBins_KernelBlackSkyAlbedo;
Float32 BinSize_KernelBlackSkyAlbedo;
UInt8 implicit_pad1[PAD_BYTES_4];
Int64 NumBins_AOT;
Float32 BinSize_AOT;
UInt8 implicit_pad2[PAD_BYTES_4];
Int64 NumKernels;
Int64 Num_Kernel_Lut;
Int64 Max_Lut_Dim;
Int64 Bpsa_Num_Bins_Solar_Zenith;
Int64 Bpsa_Num_Bins_View_Zenith;
Int64 Bpsa_Num_Bins_Rel_Azimuth;
Float32 BinCoord_SeaIce_SolarZenith[15];
Float32 BinCoord_SolarZenith[17];
Float32 BinCoord_ViewZenith[16];
Float32 BinCoord_RelAzimuth[11];
UInt8 implicit_pad3[PAD_BYTES_4];
Int64 Lut_Size_Bpsa;
Int64 Bpsa_Map_AeroModel[LSA_BPSA_MAP_AEROMODEL];
};

Now I want to use a new LUT and change to new array dimensions
Float32 BinCoord_SolarZenith[18];
Float32 BinCoord_ViewZenith[18];
Float32 BinCoord_RelAzimuth[23];

So I changed the corresponding "$ADL_HOME/xml/VIIRS/VIIRS_LSA_IP_AC_Int.xml" using text editor. However, "xmled" can not
display correctly the simple change and lost "UInt8 implicit_pad3[PAD_BYTES_4]" after "Float32 BinCoord_RelAzimuth[23];"
and report 448 bytes total length (it should be 452 bytes). If I change to "Float32 BinCoord_RelAzimuth[22];" or "Float32 BinCoord_RelAzimuth[24];",
its display became correct and "UInt8 implicit_pad3[PAD_BYTES_4]" is there. It seems that xmled has a bug and can not correctly
ingest the number 23.

If the problem ended here, that is fine since "$ADL_HOME/include/AutoGeneratedViirsProductsGbl.h" was generated. However, when I
make src library
under $ADL_HOME. It showed another error and compilation stopped

**Warning** - Struct Size Does Not Match: Scaled Error Unscaled Error
Group Name: VIIRS-LSA-IP-AC
Struct Name: IngMsdCoefficients_ViirsSurfAlbedoStruct
----------------------------------------
Source size: Map:456 Entry:452 Diff:4
Dest size: Map:456 Entry:452 Diff:4
FAIL: 1 conflicts detected in between product structure sizes and their dictionary entry definitions' sizes.

I do not know where the "MAP:456" comes from and how to resolve the conflict.

Thanks

Youhua Tang

[kbisanz]

The removal of implicit_pad3 is actually not a bug. Structure members must be on a byte boundary that is divisible by the size. So, a Int64 must be on an 8byte boundary, while a Float32 should be on a 4 byte boundary. This is a rule that the compiler uses, not something invented by ADL. In the struct, Bpsa_Num_Bins_Rel_Azimuth is on an 8 byte boundary. Then there are 59 (15+17+16+11) Float32's following it. 59*4 is not divisible by 8. This means that Int64 Lut_Size_Bpsa would *not* be on an 8 byte boundary. Therefore the compiler will behind the scenes insert 4 bytes of padding to make sure that Lut_Size_Bpsa is on an 8 byte boundary. The xmled tool (and its replacement, EBX, in ADL 4.2) automatically insert fields into the XML where the compiler will implicitly insert padding. The entire point of "UInt8 implicit_pad3[PAD_BYTES_4]" is to take up 4 bytes of space where the compiler would insert a field any way.

In the case that implicit_pad3 is removed, the reason is that after Bpsa_Num_Bins_Rel_Azimuth, there are 74 (15+18+18+23) Float32s. 74*4 is divisible by 8 so the compiler will not insert any implicit padding. Therefore, xmled will also not insert padding and implicit_pad3 is removed.

This page has more information: http://stackoverflow.com/questions/6400 ... dding-in-c

Regarding your build error, the error message is very cryptic and not something normally encountered. It is saying that one piece of the code thinks the size is 456, while another piece of code believes the size is 452. This means that something is not in sync. Normally this indicates that a library has not been rebuilt yet.

Please try doing a "make clean src library" in both $ADL_HOME/xml/VIIRS and $ADL_HOME/CMN/Utilities/Dictionary/Entries/VIIRS/src. Please post back if you still have issues.

[kbisanz]

If the case is that $ADL_HOME/xml/VIIRS/VIIRS_LSA_IP_AC_Int.xml was hand edited and UInt8 implicit_pad3[PAD_BYTES_4] is still in the XML, that is probably the issue. The reason is that after the changes to the other fields, the field implicit_pad3 was no longer needed because Lut_Size_Bpsa would now be on an 8 byte boundary. However, if the XML still has implicit_pad3 present due to hand editing, then the compiler will add an *additional* 4 bytes of pad so that Lut_Size_Bpsa is on an 8 byte boundary. I recommend editing the XML file with xmled and then rebuilding the software for changes to take effect.

After you get a successful compile you will obviously need to have a coefficient file matching the new structure. There are a couple ways to do this, but the easiest is outlined here: viewtopic.php?f=27&t=349 On step 4, there is a slight typo where the path should be $ADL_HOME/CMN/Utilities/ING/include/.

[tangyouhua]

Thank you for the 8 bytes tips. After removing the 4 bytes pad3, both issues got resolved.