HDF Unpacker

[hcronk]

For most of the NCT4 HDF5 files I have unpacked using the HDF unpacker included in ADL 3.0, the process has been smooth and the .h5 file unpacks as expected into one internal binary file and one corresponding text metadata file. However, some files (~20%?) unpack into only a metadata file without the corresponding binary file (ex: SVM01_npp_d20100910_t2126110_e2127355_b00070_c20110731000033015448_noaa_ops.h5; file ID= 53160492). What might be a reason for there to be no binary file when unpacked?

Thanks,
Heather

[kbisanz]

If you view the .h5 file, is there data in it that needs to be unpacked? It's possible that some of the items are "shell" granules, meaning they have no actual data, only metadata.

[hcronk]

The example file I mentioned looks the same as all others with every data field represented. I am not positive what the fill value is, but if all fields were filled, would that affect the unpacker's abilities? Or should it still be able to unpack into the binary and metadata as long as the arrays are there?

[kbisanz]

"SVM01" apparently is the VIIRS-M1-SDR. That product will be a "shell" output at night. A shell output is produced by some algorithms that have a mix of day only and day/night outputs. The entire algorithm can't fail because it's night, so shells are produced for day only products during night. Internal to the software, a shell is represented as a metadata collection without a binary file. When it's packaged in HDF there is a flag that indicates if the product if shell. If so, during the HDF creation process, a buffer of entirely fill data is created.

When a shell output is extracted from HDF, it's represented by a metadata collection and no binary file. The metadata collection in the .asc file should have a value of Yes for "Shell_Status". The processing code should handle it correctly. Note that having a shell granule as input may cause other algorithms to output shell granules for some of their outputs.

In conclusion, I believe you're unpacking a shell granule (indicated by the Shell_Status metadata). If so, I believe the unpacker is working correctly.

[hcronk]

Aha! That makes sense. Thank you so much for looking into it!

[ljiang]

I'm unwrapping an official ancillary file from CLASS: off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20111031_201110311200Z_20111031153623Z_ee20111031210000Z_np.h5

However I got the following error and no output files:

> tools/bin/ADL_Unpacker.exe /net/orbit159l/home/lidej/off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20111031_201110311200Z_20111031153623Z_ee20111031210000Z_np.h5

INFO 2011-10-31 17:50:49.713278 tid-47327806964464 pid-24081 (DmCoreFileTools.cpp line 2282) directory /net/orbit260l/disk1/pub/ADL3.1/ADL/data/HDF5_Unpack_Area had no files in it

Processing /net/orbit159l/home/lidej/off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20111031_201110311200Z_20111031153623Z_ee20111031210000Z_np.h5...
sh: /usr/bin/cp: No such file or directory
FATAL ERROR :
Exception Stack:
> Exception encountered in 'DDSASC_H5Unpackager.cpp', method 'parseH5ForGranules()', line 401
> Exception encountered in 'DDSASC_Granule.cpp', method 'storeGranuleToDms()', line 362
> Exception encountered in 'DDSASC_Granule.cpp', method 'unwrapData()', line 623
> STD Exception : 25InfUtil_ChecksumException (InfUtil Checksum exception: Could not open original file /disk1/pub/ADL3.1/ADL/off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20111031_201110311200Z_20111031153623Z_ee20111031210000Z_np.msd when creating the checksum)

Then I copied the h5 file to HDF5_Unpack_Area/ then the first error disappeared, but the remaining erros were the same. I noted that the 'cp' command is under "/bin/cp".

I also looked at the log file but doesn't find further details of the error.

Any ideas?

[kbisanz]

The message about ".....HDF5_Unpack_Area had no files in it" is not an error. It's just a very verbose informational message.

You're correct that it can't find the cp command. You need to update "/usr/bin/cp" to be "/bin/cp" in $ADL_HOME/CMN/Utilities/DDS/ADL_Unpacker/src/DDSASC_Granule.cpp (about line 600). Then do a "make clean library program" in $ADL_HOME/CMN/Utilities/DDS/ADL_Unpacker/src.

There will be a patch file for this, but the above code change should get you going.

[ljiang]

Thanks, now I have the file. But what is this .msd format? Is it a grib2 file or a binary file? What do i do about this .msd file? Can I use the UW's ancillary data conversion tool on this .msd file?

The message about ".....HDF5_Unpack_Area had no files in it" is not an error. It's just a very verbose informational message.

You're correct that it can't find the cp command. You need to update "/usr/bin/cp" to be "/bin/cp" in $ADL_HOME/CMN/Utilities/DDS/ADL_Unpacker/src/DDSASC_Granule.cpp (about line 600). Then do a "make clean library program" in $ADL_HOME/CMN/Utilities/DDS/ADL_Unpacker/src.

There will be a patch file for this, but the above code change should get you going.

[kbisanz]

It's a grib2 file. I'm not sure why it's named like that. I would assume UW's ancillary data conversion tool would be able to handle it.

[kbisanz]

I have emailed UW a patch for the missing cp command issue. Once they have posted it, you can apply the patch to permanently fix it.

****NOTE**** If you've previously modified CMN/Utilities/DDS/ADL_Unpacker/src/DDSASC_Granule.cpp to correct the path of cp (about line 600), it needs to be returned to original or else the patch will not apply correctly. Here's what it should look like *before* the patch:

Code: Select all

   596          // Obtain location of source data
   597          string src(file->getFullPath());
   598      
   599          // Build and issue the system command
   600          string sysCall = "/usr/bin/cp " + src + " " + finalMsdPathAndFilename;
   601          system(sysCall.c_str());

After the patch is applied, the code will check for /usr/bin/cp and /bin/cp.

The UW site should have instructions for applying the patch, but it should be:

Code: Select all

cp adl31_p1.patch $ADL_HOME
patch -p4 < adl31_p1.patch
cd $ADL_HOME/CMN/Utilities/DDS/ADL_Unpacker/src/
make clean library program