NCT4 Ancillary Data Unpack

[hcronk]

I am working to unpack and convert the NCT4 dataset to ADL input format to run the VIIRS Aerosol Algorithm. The SDRs, cloud mask, and geolocation were no problem but I have run into trouble using the ADL hdf unpacker on the NCT4 ancillary data (NCEP and NAAPS). I get the following error (same for NCEP and NAAPS fiiles):

[hcronk@orbit234l bin]$ ADL_Unpacker.exe /net/orbit234l/disk1/pub/viirs/aod/GRAVITE_Data/NCT4/ANC/NCEP/2010252/off_NCEP-GFS-24HR-ANC_GFS_NCEP_024f_20020906_201009091800Z_20020906060000Z_ee20100916180000Z_np.h5

Processing /net/orbit234l/disk1/pub/viirs/aod/GRAVITE_Data/NCT4/ANC/NCEP/2010252/off_NCEP-GFS-24HR-ANC_GFS_NCEP_024f_20020906_201009091800Z_20020906060000Z_ee20100916180000Z_np.h5...
FATAL ERROR :
Exception Stack:
> Exception encountered in 'DDSASC_H5Unpackager.cpp', method 'parseH5ForGranules()', line 388
> Exception encountered in 'DDSASC_Granule.cpp', method 'storeGranuleToDms()', line 362
> Exception encountered in 'DDSASC_GranuleMetadata.cpp', method 'setData', line 1660
> Exception encountered in 'DDSASC_GranuleMetadata.cpp', method 'getAttr', line 1396
> Exception encountered in 'DDSASC_GranuleMetadata.cpp', method 'getAttr', line 1391 : ERROR : Could not find attribute 'N_Beginning_Time_IET' in HDF file.

I'm not sure what the problem is...!

[kbisanz]

It looks like the problem is that the unpacker is expecting metadata by the name of "N_Beginning_Time_IET", but it's not present. So it throws an exception which eventually causes it to fail unpacking.

Where did you get the ancillary data? Was it all part of the NCT4 data set?

[hcronk]

Yes, all is part of NCT4 downloaded from GRAVITE/GTP

[kbisanz]

I can't quite tell from the file name. Is that NCEP data in the external or internal format? In other words, is that NCEP data an NCEP-ANC-Int file produced by Ingest?

[hcronk]

The NCEP data arrives from GRAVITE in .h5 formats for all forecast hours. I am trying to unpack each of the .h5 files to the appropriate ADL internal binary format. Is there something I need to do to the original .h5 files first to consolidate them into one per day/granule/something before using the unpacker to generate the binary files?

[kbisanz]

I don't have good news for you. The ADL unpacker *should* be able to unpack the NCEP data files. However, there appears to be a bug in the unpacker which is causing this to fail. Additionally, even after the data is extracted from HDF, it will be in a native format and must be converted to internal format. This capability won't exist officially until ADL 4.0. However, as described in this topic, viewtopic.php?f=32&t=133, the University of Wisconsin will be providing an interim solution for the native to internal format conversion problem.

As a rather difficult work around, you could use an HDF tool to dump the binary file(s) to disk and then use UW's interim solution to do the native to internal file conversion. You'd also have to create an appropriate metadata file by hand.